2024-03-11T22:37:01.994 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:37:01.994 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010 --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/orig.config.yaml 2024-03-11T22:37:02.022 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:37:02.110 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7592010' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6Gf+B0HuNRcMpFwu9v0p0gvhNIdQRSh8FzF8xoZdz6YdL2deDMWQ1xBaIdmxFbJLfIYaRxCj59ve+B6sdXIms= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBov3hdMqTycbJjxnqLPIQidB5x5HWk2MHnK2fQIe9oA68FYVPffWOxDs88jqk8+Ijd+XnhgsbvCq66DgGDsB8= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/0u5N2avfiEkrHReHX0Ot8NwHpQuoiWhyxvau7Xh9aFedNpa7PwldNySfI/O29k9ar/fMn6EcIlTz8FDJQK/A= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T22:37:02.110 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T22:37:02.111 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T22:37:02.111 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:37:02.113 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:37:02.136 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T22:37:02.136 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:37:02.136 INFO:teuthology.packaging:ref: None 2024-03-11T22:37:02.136 INFO:teuthology.packaging:tag: None 2024-03-11T22:37:02.136 INFO:teuthology.packaging:branch: squid 2024-03-11T22:37:02.136 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:37:02.136 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T22:37:02.504 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T22:37:02.506 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:37:02.513 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:37:02.513 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:37:02.530 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:37:02.540 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:37:02.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:37:02.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:47.048741', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6Gf+B0HuNRcMpFwu9v0p0gvhNIdQRSh8FzF8xoZdz6YdL2deDMWQ1xBaIdmxFbJLfIYaRxCj59ve+B6sdXIms='} 2024-03-11T22:37:02.592 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:47.052356', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBov3hdMqTycbJjxnqLPIQidB5x5HWk2MHnK2fQIe9oA68FYVPffWOxDs88jqk8+Ijd+XnhgsbvCq66DgGDsB8='} 2024-03-11T22:37:02.614 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:47.050671', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/0u5N2avfiEkrHReHX0Ot8NwHpQuoiWhyxvau7Xh9aFedNpa7PwldNySfI/O29k9ar/fMn6EcIlTz8FDJQK/A='} 2024-03-11T22:37:02.614 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:37:02.620 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T22:37:02.620 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T22:37:02.620 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-11T22:37:02.620 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:37:02.721 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f258fefaf70>, signals=[15]) 2024-03-11T22:37:02.721 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:37:02.728 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:37:02.728 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-03-11T22:37:02.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:37:02.833 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-11T22:37:02.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:37:02.920 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-11T22:37:02.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:37:03.007 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:37:03.014 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-03-11T22:37:03.033 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-03-11T22:37:03.033 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-03-11T22:37:03.090 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:37:03.091 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:37:03.092 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-03-11T22:37:03.113 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-11T22:37:03.130 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-11T22:37:03.131 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-11T22:37:03.187 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-11T22:37:03.187 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-11T22:37:03.187 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-11T22:37:03.187 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:37:03.187 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-11T22:37:03.187 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:37:03.188 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:37:03.188 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-11T22:37:03.205 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-11T22:37:03.223 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-11T22:37:03.223 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:37:03.280 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:37:03.281 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:37:03.281 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:37:03.281 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-11T22:37:03.296 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:37:03.305 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:37:03.310 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:37:03.310 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:37:03.313 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:37:03.315 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:37:03.336 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:37:03.342 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:37:03.342 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:37:03.370 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:37:03.372 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:37:03.474 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:37:03.549 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:37:03.567 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:37:03.567 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:37:03.568 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:37:03.568 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:37:03.569 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:37:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:37:03.585 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-11T22:37:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:37:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:37:03.587 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-11T22:37:03.587 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-11T22:37:03.643 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:37:03.643 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-11T22:37:03.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-11T22:37:03.645 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:37:03.645 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:37:03.645 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-361.el9.x86_64 2024-03-11T22:37:03.645 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-03-11T22:37:03.645 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-361.el9.x86_64 2024-03-11T22:37:03.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-03-11T22:37:05.783 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:37:05.847 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T22:37:05.850 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-11T22:37:07.250 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-11T22:37:07.289 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-11T22:37:07.459 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T22:37:08.246 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-11T22:37:09.758 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-11T22:37:10.839 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-11T22:37:13.157 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.0 MB 00:05 2024-03-11T22:37:17.256 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-11T22:37:18.617 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-11T22:37:18.915 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-11T22:37:22.642 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-11T22:37:24.101 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2024-03-11T22:37:26.246 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-11T22:37:30.215 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 21 MB 00:11 2024-03-11T22:37:31.160 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T22:37:31.348 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-03-11T22:37:33.350 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:37:33.350 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:37:33.436 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-11T22:37:33.449 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-11T22:37:33.450 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:37:33.451 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 100 M 2024-03-11T22:37:33.451 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-11T22:37:33.451 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-11T22:37:35.003 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 6.0 MB 00:01 2024-03-11T22:37:35.374 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-11T22:37:36.993 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-11T22:37:37.781 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:37:37.781 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:37:37.889 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-11T22:37:37.896 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-03-11T22:37:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:37:37.898 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 100 M 2024-03-11T22:37:37.898 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 123 M 2024-03-11T22:37:37.898 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-11T22:37:38.433 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 15 kB 00:00 2024-03-11T22:37:38.587 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 7.7 MB/s | 39 MB 00:05 2024-03-11T22:37:38.668 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 10 MB/s | 6.0 MB 00:00 2024-03-11T22:37:39.478 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 7.5 MB/s | 33 MB 00:04 2024-03-11T22:37:40.001 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 25 MB/s | 33 MB 00:01 2024-03-11T22:37:40.634 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 8.1 MB/s | 21 MB 00:02 2024-03-11T22:37:42.503 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-core-5.14.0-427.el9.x86_64.rpm 2.3 MB/s | 21 MB 00:08 2024-03-11T22:37:42.503 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:37:42.503 INFO:teuthology.orchestra.run.smithi116.stdout:Total 11 MB/s | 100 MB 00:09 2024-03-11T22:37:43.398 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-11T22:37:43.543 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 7.2 MB/s | 39 MB 00:05 2024-03-11T22:37:43.546 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:37:43.546 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 100 MB 00:05 2024-03-11T22:37:43.768 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-11T22:37:43.768 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-11T22:37:44.464 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-11T22:37:44.896 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-11T22:37:44.896 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-11T22:37:45.118 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-11T22:37:45.118 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-11T22:37:46.310 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-11T22:37:46.311 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-11T22:37:48.021 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-11T22:37:48.666 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:37:48.804 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:49.181 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-11T22:37:49.866 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:37:50.010 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:51.574 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:51.748 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:52.947 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:37:53.131 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:56.522 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:56.699 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:37:58.232 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:37:58.359 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:01.496 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:02.918 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 865 kB/s | 21 MB 00:24 2024-03-11T22:38:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:07.806 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T22:38:10.013 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:38:10.014 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:38:10.120 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:38:10.131 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 100 M 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 123 M 2024-03-11T22:38:10.132 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-11T22:38:10.567 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 20 MB/s | 6.0 MB 00:00 2024-03-11T22:38:10.908 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 33 MB/s | 21 MB 00:00 2024-03-11T22:38:11.459 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 33 MB/s | 39 MB 00:01 2024-03-11T22:38:11.751 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 28 MB/s | 33 MB 00:01 2024-03-11T22:38:11.751 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:38:11.751 INFO:teuthology.orchestra.run.smithi152.stdout:Total 61 MB/s | 100 MB 00:01 2024-03-11T22:38:12.630 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-11T22:38:13.040 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-11T22:38:13.040 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-11T22:38:14.408 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-11T22:38:14.409 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-11T22:38:17.350 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-11T22:38:18.005 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:38:18.138 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:38:20.932 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:38:21.126 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:38:25.859 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:38:25.987 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:30.610 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:30.610 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:38:30.610 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:38:30.696 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:30.772 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:31.100 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:38:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:38:31.101 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:38:31.529 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:38:31.530 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-11T22:38:32.003 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:32.189 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:32.189 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:32.190 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:38:32.190 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:38:32.190 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:38:32.190 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T22:38:32.190 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:38:32.190 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:32.312 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:32.313 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:32.313 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-11T22:38:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-11T22:38:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-11T22:38:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-11T22:38:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-11T22:38:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-11T22:38:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:38:32.358 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-11T22:38:32.358 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-11T22:38:32.358 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-11T22:38:32.358 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:38:32.358 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:38:32.361 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:38:32.362 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:38:32.362 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:38:32.362 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-11T22:38:32.362 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:38:32.362 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:38:32.363 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:38:32.363 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:38:32.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:38:32.980 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-11T22:38:34.210 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:34.211 INFO:teuthology.orchestra.run.smithi052.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:38:34.211 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:38:34.298 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:34.386 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:38:34.424 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-11T22:38:34.426 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-11T22:38:34.441 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.QiFvLXEf49 2024-03-11T22:38:34.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QiFvLXEf49 2024-03-11T22:38:34.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.QiFvLXEf49 2024-03-11T22:38:34.649 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.QiFvLXEf49 is 6KB 2024-03-11T22:38:34.698 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.QiFvLXEf49 2024-03-11T22:38:34.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:38:34.733 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:34.733 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:38:34.733 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:38:34.734 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:38:34.777 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:38:34.778 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:38:34.778 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:38:34.778 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:38:34.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:38:34.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-11T22:38:35.176 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:38:35.176 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:38:35.176 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-11T22:38:35.176 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:35.176 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:35.177 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:38:35.177 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:38:35.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:38:35.177 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-11T22:38:35.666 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:35.818 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:35.818 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:35.818 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:38:35.818 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:38:35.818 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:38:35.818 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T22:38:35.818 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:38:35.819 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:38:35.945 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:38:35.945 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:38:35.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-03-11T22:38:35.983 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-03-11T22:38:35.983 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-03-11T22:38:35.983 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-03-11T22:38:35.983 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:38:35.984 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:38:35.985 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:38:35.985 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-03-11T22:38:35.985 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:38:35.985 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:38:35.985 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:38:35.986 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:38:35.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:38:36.646 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-03-11T22:38:38.077 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:38:38.098 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-03-11T22:38:38.100 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-11T22:38:38.155 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.KUryZ1q33u 2024-03-11T22:38:38.155 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KUryZ1q33u 2024-03-11T22:38:38.228 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.KUryZ1q33u 2024-03-11T22:38:38.368 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.KUryZ1q33u is 6KB 2024-03-11T22:38:38.388 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.KUryZ1q33u 2024-03-11T22:38:38.402 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:38:38.470 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:38:38.471 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:38:38.471 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:38:38.471 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:38:38.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:38:38.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-03-11T22:39:00.309 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:39:00.309 INFO:teuthology.orchestra.run.smithi152.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:39:00.309 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:39:00.392 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:39:00.900 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:39:00.900 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:39:00.900 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:39:00.900 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:39:01.329 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:39:01.329 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:39:01.329 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-11T22:39:01.329 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:39:01.330 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:39:01.330 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:39:01.330 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:39:01.330 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:39:01.330 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-11T22:39:01.869 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:39:02.013 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:39:02.013 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:39:02.013 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:39:02.013 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:39:02.013 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:39:02.013 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T22:39:02.013 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:39:02.013 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:39:02.135 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:39:02.135 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:39:02.135 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-03-11T22:39:02.168 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-03-11T22:39:02.169 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:39:02.170 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:39:02.171 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:39:02.171 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:39:02.171 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:39:02.798 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-03-11T22:39:04.264 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:39:04.295 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-03-11T22:39:04.298 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-03-11T22:39:04.312 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.sBUpdMkUeB 2024-03-11T22:39:04.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sBUpdMkUeB 2024-03-11T22:39:04.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.sBUpdMkUeB 2024-03-11T22:39:04.523 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.sBUpdMkUeB is 6KB 2024-03-11T22:39:04.571 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.sBUpdMkUeB 2024-03-11T22:39:04.585 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:39:04.653 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:39:04.653 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:39:04.653 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:39:04.653 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:39:04.655 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:39:04.795 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-03-11T22:39:04.931 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T22:39:04.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-11T22:39:04.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:39:08.628 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T22:39:08.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-11T22:39:08.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:39:34.822 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T22:39:34.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-11T22:39:34.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:40:04.962 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:40:08.635 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:40:09.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-03-11T22:40:13.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-11T22:40:13.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:40:14.361 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-11T22:40:14.873 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-11T22:40:14.873 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:40:14.873 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-11T22:40:14.887 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:40:14.887 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:40:14.887 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:40:14.888 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:40:15.889 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T22:40:15.889 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:40:15.961 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-11T22:40:16.000 DEBUG:teuthology.parallel:result is None 2024-03-11T22:40:17.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-11T22:40:17.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:40:17.992 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-11T22:40:18.380 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-11T22:40:18.380 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:40:18.381 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-11T22:40:18.395 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:40:18.395 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:40:18.395 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:40:18.395 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:40:18.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-11T22:40:18.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:40:19.396 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T22:40:19.396 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:40:19.486 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-03-11T22:40:19.531 DEBUG:teuthology.parallel:result is None 2024-03-11T22:40:36.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-03-11T22:40:48.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-11T22:40:48.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:40:50.643 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-11T22:40:51.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-11T22:40:51.114 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:40:51.115 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-03-11T22:40:51.128 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:40:51.128 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:40:51.128 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:40:51.128 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:40:52.130 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T22:40:52.130 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:40:52.220 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-03-11T22:40:52.267 DEBUG:teuthology.parallel:result is None 2024-03-11T22:40:52.268 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:40:52.277 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:40:52.277 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:40:52.281 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:40:52.283 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:40:52.300 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:40:52.311 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:40:52.326 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:40:52.326 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:40:52.338 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:40:52.342 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:40:52.382 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:40:52.393 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:40:52.393 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:40:52.418 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:40:52.420 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:40:52.457 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:52.457 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:52.464 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:52.470 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:52.471 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:52.478 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:40:52.479 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:40:52.489 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:40:52.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:40:52.515 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:40:52.518 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:40:52.545 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:40:52.556 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:40:52.556 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:40:52.586 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:40:52.588 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:40:52.603 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:52.682 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:52.732 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:52.813 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:52.856 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:40:52.856 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:40:52.923 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:52.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:53.019 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:53.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:53.149 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:40:53.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:40:53.218 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:53.260 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:40:53.311 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:53.392 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:40:53.430 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:40:53.430 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:40:53.498 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-03-11T22:40:53.501 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-11T22:40:53.503 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-11T22:40:53.554 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:40:53.557 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:40:53.586 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:40:54.027 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:40:54.035 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:40:54.035 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:40:54.064 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:40:54.079 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T22:40:54.079 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-03-11T22:40:54.158 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-03-11T22:40:54.161 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-11T22:40:54.220 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-11T22:40:54.221 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-11T22:40:54.276 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-11T22:40:54.277 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:40:54.278 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-03-11T22:40:54.299 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-03-11T22:40:54.299 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-11T22:40:54.321 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-11T22:40:54.321 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-11T22:40:54.365 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-11T22:40:54.365 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:40:54.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:40:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:40:54.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:40:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:40:54.421 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:40:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:40:54.449 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:40:54.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:40:54.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:40:54.515 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:40:54.559 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:40:54.569 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:40:54.585 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:40:54.586 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi116.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T22:45:09.614 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-03-11T22:45:09.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-11T22:45:09.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:09.711 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-11T22:45:09.789 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-11T22:45:09.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-11T22:45:09.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:09.879 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-11T22:45:09.913 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-11T22:45:09.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-11T22:45:09.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:10.004 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-11T22:45:10.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-11T22:45:10.086 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T22:45:10.099 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T22:45:10.099 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:45:10.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:45:10.102 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:45:10.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:45:10.105 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:45:10.105 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:45:10.134 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:45:10.135 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:45:10.152 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:45:10.153 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:45:10.157 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:45:10.176 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:45:10.181 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-11T22:45:10.183 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-03-11T22:45:10.195 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-11T22:45:10.196 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-03-11T22:45:10.204 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-11T22:45:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:45:10.213 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:45:10.217 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-11T22:45:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:45:10.230 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:45:10.233 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:45:10.250 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:45:10.284 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:45:10.287 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:45:10.287 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-11T22:45:10.287 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.287 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.287 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.288 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:45:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:45:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-11T22:45:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.295 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.305 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:45:10.309 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:45:10.309 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-11T22:45:10.309 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.309 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.310 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.310 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:45:10.310 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T22:45:10.319 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T22:45:10.320 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi052.front.sepia.ceph.com 2024-03-11T22:45:10.320 DEBUG:teuthology.orchestra.run.smithi052:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:45:10.320 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi116.front.sepia.ceph.com 2024-03-11T22:45:10.321 DEBUG:teuthology.orchestra.run.smithi116:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:45:10.321 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi152.front.sepia.ceph.com 2024-03-11T22:45:10.322 DEBUG:teuthology.orchestra.run.smithi152:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:45:10.913 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:50 ago on Mon 11 Mar 2024 10:44:20 PM UTC. 2024-03-11T22:45:10.918 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 10:44:22 PM UTC. 2024-03-11T22:45:10.962 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:49 ago on Mon 11 Mar 2024 10:44:21 PM UTC. 2024-03-11T22:45:11.059 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-11T22:45:11.069 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-11T22:45:11.070 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:45:11.071 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-03-11T22:45:11.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:45:11.079 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.1 M 2024-03-11T22:45:11.079 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 9.8 M 2024-03-11T22:45:11.079 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-11T22:45:11.086 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.1 M 2024-03-11T22:45:11.086 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 9.8 M 2024-03-11T22:45:11.086 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-11T22:45:11.108 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-11T22:45:11.109 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-11T22:45:11.109 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-11T22:45:11.109 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-03-11T22:45:11.110 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:45:11.133 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.1 M 2024-03-11T22:45:11.133 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 9.8 M 2024-03-11T22:45:11.133 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-11T22:45:11.620 INFO:teuthology.orchestra.run.smithi052.stdout:runc-1.1.12-2.el9.x86_64.rpm 13 MB/s | 3.1 MB 00:00 2024-03-11T22:45:11.620 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:11.620 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.9 MB/s | 3.1 MB 00:00 2024-03-11T22:45:11.656 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-11T22:45:11.662 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-11T22:45:11.662 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-11T22:45:11.744 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-11T22:45:11.744 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-11T22:45:11.897 INFO:teuthology.orchestra.run.smithi116.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.0 MB/s | 3.1 MB 00:00 2024-03-11T22:45:11.897 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:11.897 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.8 MB/s | 3.1 MB 00:00 2024-03-11T22:45:11.933 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-11T22:45:11.939 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-11T22:45:11.939 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-11T22:45:11.954 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-11T22:45:12.018 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-11T22:45:12.018 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-11T22:45:12.058 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:12.076 INFO:teuthology.orchestra.run.smithi152.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.9 MB/s | 3.1 MB 00:00 2024-03-11T22:45:12.076 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:45:12.076 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.3 MB/s | 3.1 MB 00:00 2024-03-11T22:45:12.112 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-11T22:45:12.118 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-11T22:45:12.118 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-11T22:45:12.197 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-11T22:45:12.197 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-11T22:45:12.238 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-11T22:45:12.333 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:12.415 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-11T22:45:12.503 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:12.646 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:12.862 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:12.945 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:45:12.946 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-11T22:45:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:45:12.946 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:45:12.946 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-11T22:45:13.177 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:13.180 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:13.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:45:13.180 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-11T22:45:13.180 INFO:teuthology.orchestra.run.smithi116.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:45:13.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-11T22:45:13.180 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-11T22:45:13.220 DEBUG:teuthology.parallel:result is None 2024-03-11T22:45:13.413 DEBUG:teuthology.parallel:result is None 2024-03-11T22:45:13.503 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:45:13.503 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:45:13.503 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-11T22:45:13.503 INFO:teuthology.orchestra.run.smithi152.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:45:13.503 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:45:13.504 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-11T22:45:13.745 DEBUG:teuthology.parallel:result is None 2024-03-11T22:45:13.746 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T22:45:13.756 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T22:45:13.756 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:45:13.756 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:45:13.774 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:45:13.774 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-03-11T22:45:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:45:13.829 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:13.829 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:45:13.829 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:13.829 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:13.829 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:45:08.506921302 +0000 2024-03-11T22:45:13.830 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:44:46.862188029 +0000 2024-03-11T22:45:13.830 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:44:46.862188029 +0000 2024-03-11T22:45:13.830 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:44:46.862188029 +0000 2024-03-11T22:45:13.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:45:13.895 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:45:13.895 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:45:13.895 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.4222e-05 s, 6.1 MB/s 2024-03-11T22:45:13.896 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:45:13.952 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:45:08.506921302 +0000 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:44:47.374181720 +0000 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:44:47.374181720 +0000 2024-03-11T22:45:14.007 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:44:47.374181720 +0000 2024-03-11T22:45:14.008 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:45:14.073 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:45:14.073 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:45:14.073 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 7.8407e-05 s, 6.5 MB/s 2024-03-11T22:45:14.074 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:45:14.131 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-03-11T22:45:14.186 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:45:14.187 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:14.187 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T22:45:14.187 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:14.187 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:14.187 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:45:08.506921302 +0000 2024-03-11T22:45:14.187 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:44:47.698177728 +0000 2024-03-11T22:45:14.188 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:44:47.698177728 +0000 2024-03-11T22:45:14.188 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:44:47.698177728 +0000 2024-03-11T22:45:14.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:45:14.253 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:45:14.253 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:45:14.254 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.8033e-05 s, 5.2 MB/s 2024-03-11T22:45:14.255 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:45:14.310 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:45:08.506921302 +0000 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:44:48.039173525 +0000 2024-03-11T22:45:14.365 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:44:48.039173525 +0000 2024-03-11T22:45:14.366 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:44:48.039173525 +0000 2024-03-11T22:45:14.366 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:45:14.429 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:45:14.430 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:45:14.430 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.243e-05 s, 5.5 MB/s 2024-03-11T22:45:14.431 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:45:14.486 DEBUG:teuthology.orchestra.run.smithi052:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:45:14.641 INFO:teuthology.orchestra.run.smithi052.stdout:loop 2024-03-11T22:45:14.643 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_1... 2024-03-11T22:45:14.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:45:14.680 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-11T22:45:14.717 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:45:14.747 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_2... 2024-03-11T22:45:14.747 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:45:14.825 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-11T22:45:14.863 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:45:14.898 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_3... 2024-03-11T22:45:14.898 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:45:14.976 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-11T22:45:15.013 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:45:15.048 INFO:tasks.nvme_loop:Connecting nvme_loop smithi052:/dev/vg_nvme/lv_4... 2024-03-11T22:45:15.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:45:15.132 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-11T22:45:15.169 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:45:15.197 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:45:15.197 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:45:15.252 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme list 2024-03-11T22:45:15.317 INFO:teuthology.orchestra.run.smithi052.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:45:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:45:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme0n1 PHMB7513010N480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-11T22:45:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme1n1 b6ba118092571df5e527 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme2n1 dc2184990ddde91cdd34 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme3n1 19feec2ada5b97b59cc7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:15.318 INFO:teuthology.orchestra.run.smithi052.stdout:/dev/nvme4n1 3458ba7a8a84a78a5817 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:15.319 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:45:15.319 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:45:15.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/scratch_devs 2024-03-11T22:45:15.386 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:45:15.386 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:45:15.405 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:45:15.405 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:45:08.515100324 +0000 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:44:45.839449117 +0000 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:44:45.839449117 +0000 2024-03-11T22:45:15.463 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:44:45.839449117 +0000 2024-03-11T22:45:15.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:45:15.532 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:45:15.532 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:45:15.532 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000194379 s, 2.6 MB/s 2024-03-11T22:45:15.534 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:45:15.592 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:45:08.515100324 +0000 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:44:46.181443857 +0000 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:44:46.181443857 +0000 2024-03-11T22:45:15.651 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:44:46.181443857 +0000 2024-03-11T22:45:15.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:45:15.720 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:45:15.720 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:45:15.720 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000188848 s, 2.7 MB/s 2024-03-11T22:45:15.721 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:45:15.781 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-11T22:45:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:45:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:15.839 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T22:45:15.840 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:15.840 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:15.840 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:45:08.515100324 +0000 2024-03-11T22:45:15.840 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:44:46.508438827 +0000 2024-03-11T22:45:15.840 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:44:46.508438827 +0000 2024-03-11T22:45:15.840 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:44:46.508438827 +0000 2024-03-11T22:45:15.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:45:15.906 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:45:15.906 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:45:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000231994 s, 2.2 MB/s 2024-03-11T22:45:15.908 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:45:15.965 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-11T22:45:16.020 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:45:16.020 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:45:08.515100324 +0000 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:44:46.847433613 +0000 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:44:46.847433613 +0000 2024-03-11T22:45:16.021 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:44:46.847433613 +0000 2024-03-11T22:45:16.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:45:16.087 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:45:16.087 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:45:16.087 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000188621 s, 2.7 MB/s 2024-03-11T22:45:16.088 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:45:16.146 DEBUG:teuthology.orchestra.run.smithi116:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:45:16.322 INFO:teuthology.orchestra.run.smithi116.stdout:loop 2024-03-11T22:45:16.324 INFO:tasks.nvme_loop:Connecting nvme_loop smithi116:/dev/vg_nvme/lv_1... 2024-03-11T22:45:16.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:45:16.362 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-11T22:45:16.399 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:45:16.428 INFO:tasks.nvme_loop:Connecting nvme_loop smithi116:/dev/vg_nvme/lv_2... 2024-03-11T22:45:16.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:45:16.507 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-11T22:45:16.547 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:45:16.575 INFO:tasks.nvme_loop:Connecting nvme_loop smithi116:/dev/vg_nvme/lv_3... 2024-03-11T22:45:16.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:45:16.655 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-11T22:45:16.694 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:45:16.731 INFO:tasks.nvme_loop:Connecting nvme_loop smithi116:/dev/vg_nvme/lv_4... 2024-03-11T22:45:16.731 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:45:16.810 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-11T22:45:16.853 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:45:16.893 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:45:16.894 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:45:16.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo nvme list 2024-03-11T22:45:17.019 INFO:teuthology.orchestra.run.smithi116.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:45:17.019 INFO:teuthology.orchestra.run.smithi116.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:45:17.019 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/nvme0n1 PHFT620400VJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:45:17.019 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/nvme1n1 99d57a0f0300ab9bcc18 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:17.020 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/nvme2n1 ee4d1a537ce9016e5459 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:17.020 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/nvme3n1 cb9cb1c9415af4464ef7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:17.020 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/nvme4n1 8d2640b8d59fb4b30626 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:17.021 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:45:17.021 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:45:17.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/scratch_devs 2024-03-11T22:45:17.088 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:45:17.088 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:45:17.107 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:45:17.107 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-11T22:45:17.164 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:45:17.164 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:45:08.530378370 +0000 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:44:46.500642876 +0000 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:44:46.500642876 +0000 2024-03-11T22:45:17.165 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:44:46.500642876 +0000 2024-03-11T22:45:17.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:45:17.234 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:45:17.235 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:45:17.235 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000232255 s, 2.2 MB/s 2024-03-11T22:45:17.236 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:45:17.294 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-11T22:45:17.351 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:45:08.530378370 +0000 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:44:46.849638686 +0000 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:44:46.849638686 +0000 2024-03-11T22:45:17.352 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:44:46.849638686 +0000 2024-03-11T22:45:17.352 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:45:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:45:17.420 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:45:17.421 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000163115 s, 3.1 MB/s 2024-03-11T22:45:17.422 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:45:17.479 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:45:08.530378370 +0000 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:44:47.196634520 +0000 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:44:47.196634520 +0000 2024-03-11T22:45:17.535 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:44:47.196634520 +0000 2024-03-11T22:45:17.536 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:45:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:45:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:45:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000183671 s, 2.8 MB/s 2024-03-11T22:45:17.603 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:45:17.659 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-11T22:45:17.716 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:45:17.716 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:45:17.716 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:45:17.716 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:45:17.717 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:45:17.717 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:45:08.530378370 +0000 2024-03-11T22:45:17.717 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:44:47.557630185 +0000 2024-03-11T22:45:17.717 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:44:47.557630185 +0000 2024-03-11T22:45:17.717 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:44:47.557630185 +0000 2024-03-11T22:45:17.718 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:45:17.783 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:45:17.783 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:45:17.783 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000185084 s, 2.8 MB/s 2024-03-11T22:45:17.785 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:45:17.841 DEBUG:teuthology.orchestra.run.smithi152:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:45:17.991 INFO:teuthology.orchestra.run.smithi152.stdout:loop 2024-03-11T22:45:17.992 INFO:tasks.nvme_loop:Connecting nvme_loop smithi152:/dev/vg_nvme/lv_1... 2024-03-11T22:45:17.992 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:45:18.030 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-03-11T22:45:18.067 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:45:18.096 INFO:tasks.nvme_loop:Connecting nvme_loop smithi152:/dev/vg_nvme/lv_2... 2024-03-11T22:45:18.096 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:45:18.187 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-03-11T22:45:18.227 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:45:18.257 INFO:tasks.nvme_loop:Connecting nvme_loop smithi152:/dev/vg_nvme/lv_3... 2024-03-11T22:45:18.257 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:45:18.307 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-03-11T22:45:18.347 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:45:18.376 INFO:tasks.nvme_loop:Connecting nvme_loop smithi152:/dev/vg_nvme/lv_4... 2024-03-11T22:45:18.376 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:45:18.454 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-03-11T22:45:18.493 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:45:18.524 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:45:18.524 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:45:18.580 DEBUG:teuthology.orchestra.run.smithi152:> sudo nvme list 2024-03-11T22:45:18.646 INFO:teuthology.orchestra.run.smithi152.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:45:18.646 INFO:teuthology.orchestra.run.smithi152.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:45:18.647 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/nvme0n1 CVFT623300JT400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:45:18.647 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/nvme1n1 310f4811b8f1d6273564 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:18.647 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/nvme2n1 1718228592a696eb2c5e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:18.647 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/nvme3n1 f20c140db9425dfdb8e7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:18.647 INFO:teuthology.orchestra.run.smithi152.stdout:/dev/nvme4n1 bd9aa492a179a6ea3c4e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:45:18.648 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:45:18.648 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:45:18.648 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/scratch_devs 2024-03-11T22:45:18.715 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T22:45:18.821 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T22:45:18.821 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:45:18.822 INFO:tasks.cephadm:Cluster fsid is 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:45:18.822 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T22:45:18.822 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.52', 'mon.b': '172.21.15.116', 'mon.c': '172.21.15.152'} 2024-03-11T22:45:18.822 INFO:tasks.cephadm:First mon is mon.a on smithi052 2024-03-11T22:45:18.822 INFO:tasks.cephadm:First mgr is y 2024-03-11T22:45:18.822 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T22:45:18.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo hostname $(hostname -s) 2024-03-11T22:45:18.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-11T22:45:18.890 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-03-11T22:45:18.917 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T22:45:18.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:45:19.068 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T22:45:19.207 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:45:19.209 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T22:45:19.209 DEBUG:teuthology.orchestra.run.smithi052:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:20.096 INFO:teuthology.orchestra.run.smithi052.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:45 /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:20.096 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:21.226 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:45 /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:21.226 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:22.342 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:45 /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:22.342 DEBUG:teuthology.orchestra.run.smithi052:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:22.362 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:22.381 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:45:22.415 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T22:45:22.415 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:45:22.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:45:22.423 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:45:23.247 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:45:23.248 INFO:teuthology.orchestra.run.smithi052.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:45:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:46:18.999 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-11T22:46:18.999 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:46:18.999 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:46:18.999 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-11T22:46:19.000 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:46:19.000 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-11T22:46:19.000 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-11T22:46:25.901 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-11T22:46:25.902 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:46:25.902 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:46:25.902 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-03-11T22:46:25.902 INFO:teuthology.orchestra.run.smithi152.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:46:25.902 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-03-11T22:46:25.902 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-11T22:46:26.188 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2024-03-11T22:46:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:46:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:46:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "repo_digests": [ 2024-03-11T22:46:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:46:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2024-03-11T22:46:26.189 INFO:teuthology.orchestra.run.smithi052.stdout:} 2024-03-11T22:46:26.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph 2024-03-11T22:46:26.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-11T22:46:26.305 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-03-11T22:46:26.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /etc/ceph 2024-03-11T22:46:26.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-11T22:46:26.436 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-03-11T22:46:26.463 INFO:tasks.cephadm:Writing seed config... 2024-03-11T22:46:26.465 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T22:46:26.465 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T22:46:26.465 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-11T22:46:26.465 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-11T22:46:26.465 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T22:46:26.465 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T22:46:26.466 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T22:46:26.466 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T22:46:26.466 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T22:46:26.466 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T22:46:26.466 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:46:26.467 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T22:46:26.482 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 09234c40-dff9-11ee-95c8-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T22:46:26.482 DEBUG:teuthology.orchestra.run.smithi052:mon.a> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T22:46:26.525 DEBUG:teuthology.orchestra.run.smithi052:mgr.y> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T22:46:26.568 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T22:46:26.569 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.52 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:46:26.764 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:46:26.764 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '09234c40-dff9-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.52', '--skip-admin-label'] 2024-03-11T22:46:26.787 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:46:26.788 INFO:teuthology.orchestra.run.smithi052.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T22:46:26.788 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying podman|docker is present... 2024-03-11T22:46:26.812 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:46:26.812 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying lvm2 is present... 2024-03-11T22:46:26.812 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying time synchronization is in place... 2024-03-11T22:46:26.819 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:46:26.819 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:46:26.825 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:46:26.826 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-11T22:46:26.831 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-03-11T22:46:26.837 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-03-11T22:46:26.838 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:46:26.838 INFO:teuthology.orchestra.run.smithi052.stdout:Repeating the final host check... 2024-03-11T22:46:26.861 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:46:26.861 INFO:teuthology.orchestra.run.smithi052.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T22:46:26.861 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl is present 2024-03-11T22:46:26.861 INFO:teuthology.orchestra.run.smithi052.stdout:lvcreate is present 2024-03-11T22:46:26.866 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:46:26.866 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:46:26.872 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:46:26.872 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-11T22:46:26.878 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout enabled 2024-03-11T22:46:26.885 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout active 2024-03-11T22:46:26.885 INFO:teuthology.orchestra.run.smithi052.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:46:26.885 INFO:teuthology.orchestra.run.smithi052.stdout:Host looks OK 2024-03-11T22:46:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Cluster fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Acquiring lock 139841381227104 on /run/cephadm/09234c40-dff9-11ee-95c8-87774f69a715.lock 2024-03-11T22:46:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Lock 139841381227104 acquired on /run/cephadm/09234c40-dff9-11ee-95c8-87774f69a715.lock 2024-03-11T22:46:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 3300 ... 2024-03-11T22:46:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying IP 172.21.15.52 port 6789 ... 2024-03-11T22:46:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Base mon IP(s) is [172.21.15.52:3300, 172.21.15.52:6789], mon addrv is [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-03-11T22:46:26.888 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.52 metric 100 2024-03-11T22:46:26.888 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.52 metric 100 2024-03-11T22:46:26.891 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T22:46:26.891 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T22:46:26.891 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T22:46:26.894 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T22:46:26.894 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T22:46:26.894 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:46:26.895 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T22:46:26.895 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15de/64 scope link noprefixroute 2024-03-11T22:46:26.895 INFO:teuthology.orchestra.run.smithi052.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:46:26.895 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-03-11T22:46:26.895 INFO:teuthology.orchestra.run.smithi052.stdout:Mon IP `172.21.15.52` is in CIDR network `172.21.0.0/20` 2024-03-11T22:46:26.895 INFO:teuthology.orchestra.run.smithi052.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T22:46:26.896 INFO:teuthology.orchestra.run.smithi052.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T22:46:26.896 INFO:teuthology.orchestra.run.smithi052.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:46:27.300 INFO:teuthology.orchestra.run.smithi052.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T22:46:28.253 INFO:teuthology.orchestra.run.smithi052.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:46:28.253 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:46:28.253 INFO:teuthology.orchestra.run.smithi052.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T22:46:30.293 INFO:teuthology.orchestra.run.smithi052.stdout:stat: stdout 167 167 2024-03-11T22:46:30.293 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial keys... 2024-03-11T22:46:38.606 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQDJie9l+Iq1CBAAOwrHydp4ZvJkQLWglBkFFA== 2024-03-11T22:46:39.519 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQDOie9ljW48NxAANhyzOoy5GoMuGJXnk6c+IA== 2024-03-11T22:46:40.392 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph-authtool: stdout AQDPie9l+8icNBAADaRUczciBgniHbbAkPkK/A== 2024-03-11T22:46:40.393 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial monmap... 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool for a [v2:172.21.15.52:3300,v1:172.21.15.52:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: set fsid to 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:41.447 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:46:41.448 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:46:41.448 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mon... 2024-03-11T22:46:42.301 INFO:teuthology.orchestra.run.smithi052.stdout:create mon.a on 2024-03-11T22:46:42.670 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T22:46:42.970 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-09234c40-dff9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-09234c40-dff9-11ee-95c8-87774f69a715.target. 2024-03-11T22:46:42.970 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-09234c40-dff9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-09234c40-dff9-11ee-95c8-87774f69a715.target. 2024-03-11T22:46:43.608 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:46:43.608 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T22:46:43.767 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-09234c40-dff9-11ee-95c8-87774f69a715.target.wants/ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-09234c40-dff9-11ee-95c8-87774f69a715@.service. 2024-03-11T22:46:44.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 podman[25312]: 2024-03-11 22:46:43.911929343 +0000 UTC m=+0.017650067 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:46:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:46:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-11T22:46:44.301 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:46:44.302 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-11T22:46:44.302 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-11T22:46:44.302 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:46:44.302 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon to start... 2024-03-11T22:46:44.302 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mon... 2024-03-11T22:46:44.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 podman[25312]: 2024-03-11 22:46:44.201152207 +0000 UTC m=+0.306872944 container init 106ca9f16e68b6c46b0a25892dd54f15eb08db04f821f29d2c83ced63f068eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:46:44.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 podman[25312]: 2024-03-11 22:46:44.206153837 +0000 UTC m=+0.311874561 container start 106ca9f16e68b6c46b0a25892dd54f15eb08db04f821f29d2c83ced63f068eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T22:46:44.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 ceph-mon[25351]: mkfs 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:44.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 bash[25312]: 106ca9f16e68b6c46b0a25892dd54f15eb08db04f821f29d2c83ced63f068eaf 2024-03-11T22:46:44.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 ceph-mon[25351]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:46:44.538 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:44 smithi052 systemd[1]: Started Ceph mon.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:46:45.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:45 smithi052 ceph-mon[25351]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:46:45.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:45 smithi052 ceph-mon[25351]: monmap e1: 1 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:46:45.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:45 smithi052 ceph-mon[25351]: fsmap 2024-03-11T22:46:45.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:45 smithi052 ceph-mon[25351]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:46:45.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:45 smithi052 ceph-mon[25351]: mgrmap e1: no daemons active 2024-03-11T22:46:45.439 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:45 smithi052 ceph-mon[25351]: from='client.? 172.21.15.52:0/4163082365' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T22:46:45.442 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout id: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout services: 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.571443s) 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout data: 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T22:46:45.443 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:45.444 INFO:teuthology.orchestra.run.smithi052.stdout:mon is available 2024-03-11T22:46:45.444 INFO:teuthology.orchestra.run.smithi052.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T22:46:46.536 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:46 smithi052 ceph-mon[25351]: from='client.? 172.21.15.52:0/2983688869' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:46:46.536 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:46 smithi052 ceph-mon[25351]: from='client.? 172.21.15.52:0/2983688869' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.52:3300,v1:172.21.15.52:6789] 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:46:46.540 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:46:46.541 INFO:teuthology.orchestra.run.smithi052.stdout:Generating new minimal ceph.conf... 2024-03-11T22:46:47.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:47 smithi052 ceph-mon[25351]: from='client.? 172.21.15.52:0/2421004768' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:46:47.577 INFO:teuthology.orchestra.run.smithi052.stdout:Restarting the monitor... 2024-03-11T22:46:47.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:47 smithi052 systemd[1]: Stopping Ceph mon.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:46:48.237 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:47 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a[25328]: 2024-03-11T22:46:47.823+0000 7f4ddbf09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:46:48.237 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:47 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a[25328]: 2024-03-11T22:46:47.823+0000 7f4ddbf09700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T22:46:48.237 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:47 smithi052 podman[25670]: 2024-03-11 22:46:47.998032961 +0000 UTC m=+0.266331434 container died 106ca9f16e68b6c46b0a25892dd54f15eb08db04f821f29d2c83ced63f068eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:46:48.528 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 podman[25670]: 2024-03-11 22:46:48.296121143 +0000 UTC m=+0.564419657 container cleanup 106ca9f16e68b6c46b0a25892dd54f15eb08db04f821f29d2c83ced63f068eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T22:46:48.528 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 bash[25670]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:46:48.778 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 podman[25690]: 2024-03-11 22:46:48.521766467 +0000 UTC m=+0.514829420 container remove 106ca9f16e68b6c46b0a25892dd54f15eb08db04f821f29d2c83ced63f068eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:46:49.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T22:46:49.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 systemd[1]: Stopped Ceph mon.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:46:49.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 systemd[1]: Starting Ceph mon.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:46:49.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 podman[25785]: 2024-03-11 22:46:48.964097927 +0000 UTC m=+0.091949753 container create d0af2840eadd0c8cde49aa1e74191719a7ce460dc5f180c9dcfdb08125c8bdfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-11T22:46:49.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:48 smithi052 podman[25785]: 2024-03-11 22:46:48.889665736 +0000 UTC m=+0.017517563 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:46:49.259 INFO:teuthology.orchestra.run.smithi052.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T22:46:49.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 podman[25785]: 2024-03-11 22:46:49.157922881 +0000 UTC m=+0.285774710 container init d0af2840eadd0c8cde49aa1e74191719a7ce460dc5f180c9dcfdb08125c8bdfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:46:49.468 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 podman[25785]: 2024-03-11 22:46:49.163664911 +0000 UTC m=+0.291516739 container start d0af2840eadd0c8cde49aa1e74191719a7ce460dc5f180c9dcfdb08125c8bdfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: pidfile_write: ignore empty --pid-file 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: load: jerasure load: lrc 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Git sha 0 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: DB SUMMARY 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: DB Session ID: 4BDTSFS3P38TJ7E9GBR9 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T22:46:49.469 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84629 ; 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.env: 0x55f83279dae0 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.info_log: 0x55f834e9f8c0 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.statistics: (nil) 2024-03-11T22:46:49.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.use_fsync: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.db_log_dir: 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.wal_dir: 2024-03-11T22:46:49.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.write_buffer_manager: 0x55f834e73400 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:46:49.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.unordered_write: 0 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.row_cache: None 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.wal_filter: None 2024-03-11T22:46:49.473 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:46:49.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:46:49.475 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.wal_compression: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:46:49.476 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:46:49.477 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_open_files: -1 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Compression algorithms supported: 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kZSTD supported: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:46:49.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.merge_operator: 2024-03-11T22:46:49.479 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_filter: None 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f834e9f9c0) 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_type: 0 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_index_type: 0 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_shortening: 1 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: checksum: 4 2024-03-11T22:46:49.480 INFO:journalctl@ceph.mon.a.smithi052.stdout: no_block_cache: 0 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache: 0x55f834ea3090 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_options: 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: capacity : 536870912 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_shard_bits : 4 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: strict_capacity_limit : 0 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_cache_compressed: (nil) 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: persistent_cache: (nil) 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size: 4096 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_size_deviation: 10 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_restart_interval: 16 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: index_block_restart_interval: 1 2024-03-11T22:46:49.481 INFO:journalctl@ceph.mon.a.smithi052.stdout: metadata_block_size: 4096 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: partition_filters: 0 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: use_delta_encoding: 1 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: filter_policy: bloomfilter 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: whole_key_filtering: 1 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: verify_compression: 0 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: format_version: 5 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_index_compression: 1 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: block_align: 0 2024-03-11T22:46:49.482 INFO:journalctl@ceph.mon.a.smithi052.stdout: max_auto_readahead_size: 262144 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout: prepopulate_block_cache: 0 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression: NoCompression 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.num_levels: 7 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:46:49.483 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:46:49.484 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:46:49.485 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:46:49.486 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:46:49.487 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:46:49.488 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.ttl: 2592000 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:46:49.489 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6129c510-0043-46f2-aaec-968ba715f056 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197209192826, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T22:46:49.490 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197209193675, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81254, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79441, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73750, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710197209, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6129c510-0043-46f2-aaec-968ba715f056", "db_session_id": "4BDTSFS3P38TJ7E9GBR9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197209193759, "job": 1, "event": "recovery_finished"} 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f834f94000 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: DB pointer 0x55f834f7e000 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** DB Stats ** 2024-03-11T22:46:49.491 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: L0 2/0 81.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Sum 2/0 81.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:46:49.492 INFO:journalctl@ceph.mon.a.smithi052.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** Compaction Stats [default] ** 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:46:49.493 INFO:journalctl@ceph.mon.a.smithi052.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: Cumulative compaction: 0.00 GB write, 26.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: Interval compaction: 0.00 GB write, 26.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache BinnedLRUCache@0x55f834ea3090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: starting mon.a rank 0 at public addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] at bind addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???) e1 preinit fsid 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).mds e1 new map 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).mds e1 print_map 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: e1 2024-03-11T22:46:49.494 INFO:journalctl@ceph.mon.a.smithi052.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout: legacy client fscid: -1 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout: 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout: No filesystems configured 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:46:49.495 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 bash[25785]: d0af2840eadd0c8cde49aa1e74191719a7ce460dc5f180c9dcfdb08125c8bdfd 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 systemd[1]: Started Ceph mon.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: monmap e1: 1 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: fsmap 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:46:49.496 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:49 smithi052 ceph-mon[25824]: mgrmap e1: no daemons active 2024-03-11T22:46:50.306 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T22:46:50.328 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:46:50.328 INFO:teuthology.orchestra.run.smithi052.stdout:Creating mgr... 2024-03-11T22:46:50.328 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T22:46:50.328 INFO:teuthology.orchestra.run.smithi052.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T22:46:50.582 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:46:50.582 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Failed to reset failed state of unit ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T22:46:50.736 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-09234c40-dff9-11ee-95c8-87774f69a715.target.wants/ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-09234c40-dff9-11ee-95c8-87774f69a715@.service. 2024-03-11T22:46:51.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:50 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/887669325' entity='client.admin' 2024-03-11T22:46:51.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:50 smithi052 podman[26062]: 2024-03-11 22:46:50.973207136 +0000 UTC m=+0.127421402 container create afc4e2cd73382a16ce27c701dfb21488e08b4d4c393f15c77f6bc17f36200c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T22:46:51.330 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:46:51.330 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-11T22:46:51.338 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:46:51.339 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-11T22:46:51.339 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-11T22:46:51.339 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:46:51.348 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:46:51.349 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-11T22:46:51.356 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:46:51.356 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-11T22:46:51.356 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-11T22:46:51.356 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T22:46:51.356 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr to start... 2024-03-11T22:46:51.356 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr... 2024-03-11T22:46:51.481 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:51 smithi052 podman[26062]: 2024-03-11 22:46:51.200620315 +0000 UTC m=+0.354834580 container init afc4e2cd73382a16ce27c701dfb21488e08b4d4c393f15c77f6bc17f36200c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:46:51.481 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:51 smithi052 podman[26062]: 2024-03-11 22:46:51.205666549 +0000 UTC m=+0.359880818 container start afc4e2cd73382a16ce27c701dfb21488e08b4d4c393f15c77f6bc17f36200c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-03-11T22:46:51.481 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:51 smithi052 bash[26062]: afc4e2cd73382a16ce27c701dfb21488e08b4d4c393f15c77f6bc17f36200c8c 2024-03-11T22:46:51.481 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:51 smithi052 systemd[1]: Started Ceph mgr.y for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:46:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:52 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3594596009' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:46:52.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:51.773+0000 7f8cf3c1e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:46:52.073 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:51.870+0000 7f8cf3c1e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:46:52.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:52.541+0000 7f8cf3c1e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:46:52.584 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "09234c40-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:46:52.585 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:46:52.586 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.590 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:46:52.591 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:46:44.273485+0000", 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:46:52.592 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (1/15)... 2024-03-11T22:46:52.882 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:52.636+0000 7f8cf3c1e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:46:52.883 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:52.778+0000 7f8cf3c1e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:46:52.883 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:52.875+0000 7f8cf3c1e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:46:53.232 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:52.959+0000 7f8cf3c1e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:46:53.232 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:53.224+0000 7f8cf3c1e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:46:53.525 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:53.335+0000 7f8cf3c1e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:46:53.525 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:53.422+0000 7f8cf3c1e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:46:53.525 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:53.517+0000 7f8cf3c1e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:46:53.822 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:53.694+0000 7f8cf3c1e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:46:54.949 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:54 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:54.686+0000 7f8cf3c1e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:46:54.949 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:54 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:54.821+0000 7f8cf3c1e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:46:55.268 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:55 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/872406553' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:46:55.268 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:55.169+0000 7f8cf3c1e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:46:55.268 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:55.260+0000 7f8cf3c1e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:46:55.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:55.461+0000 7f8cf3c1e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:46:55.674 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "09234c40-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:46:55.675 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:46:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:46:55.680 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.680 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:46:55.680 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:46:55.681 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:46:55.682 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:55.683 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:46:44.273485+0000", 2024-03-11T22:46:55.683 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:46:55.683 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:55.683 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:46:55.683 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:46:55.683 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (2/15)... 2024-03-11T22:46:55.942 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:55.648+0000 7f8cf3c1e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:46:56.203 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:55.935+0000 7f8cf3c1e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:46:56.204 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:56.196+0000 7f8cf3c1e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:46:56.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:56.433+0000 7f8cf3c1e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:46:57.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:57 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:57.257+0000 7f8cf3c1e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:46:58.046 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:57 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:57.841+0000 7f8cf3c1e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:46:58.046 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:57 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:57.944+0000 7f8cf3c1e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:46:58.322 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:46:58.166+0000 7f8cf3c1e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: Activating manager daemon y 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: mgrmap e2: y(active, starting, since 0.00334735s) 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: Manager daemon y is now available 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' 2024-03-11T22:46:58.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='mgr.14100 172.21.15.52:0/1831137673' entity='mgr.y' 2024-03-11T22:46:58.324 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:46:58 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/4010654367' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "09234c40-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:46:58.874 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:58.875 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.876 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:46:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:46:58.878 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:46:58.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:46:44.273485+0000", 2024-03-11T22:46:58.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:46:58.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:46:58.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:46:58.879 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:46:58.879 INFO:teuthology.orchestra.run.smithi052.stdout:mgr not available, waiting (3/15)... 2024-03-11T22:47:00.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:00 smithi052 ceph-mon[25824]: mgrmap e3: y(active, since 1.00734s) 2024-03-11T22:47:02.236 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsid": "09234c40-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:47:02.237 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:47:02.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:47:02.239 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:47:02.242 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:47:02.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:47:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:47:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:46:44.273485+0000", 2024-03-11T22:47:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:47:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:47:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:47:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:47:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:mgr is available 2024-03-11T22:47:02.449 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:02 smithi052 ceph-mon[25824]: mgrmap e4: y(active, since 3s) 2024-03-11T22:47:02.450 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:02 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/43661486' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:47:03.470 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:03 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/244180425' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:47:03.471 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:03 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/244180425' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:47:03.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:47:03.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:47:03.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout fsid = 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:47:03.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:47:03.474 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:47:03.475 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling cephadm module... 2024-03-11T22:47:04.567 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:04 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2597754352' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T22:47:04.568 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:04 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ignoring --setuser ceph since I am not root 2024-03-11T22:47:04.568 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:04 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ignoring --setgroup ceph since I am not root 2024-03-11T22:47:05.195 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:04 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:04.844+0000 7f57d4542200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:47:05.196 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:04 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:04.941+0000 7f57d4542200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:47:05.567 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:05 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2597754352' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T22:47:05.567 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:05 smithi052 ceph-mon[25824]: mgrmap e5: y(active, since 6s) 2024-03-11T22:47:05.822 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:05.619+0000 7f57d4542200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:47:05.822 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:05.714+0000 7f57d4542200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:47:06.240 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:05.858+0000 7f57d4542200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:47:06.240 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:05.956+0000 7f57d4542200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:47:06.241 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:06.040+0000 7f57d4542200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:47:06.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:47:06.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T22:47:06.243 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:47:06.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:47:06.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:47:06.244 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:47:06.244 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-03-11T22:47:06.244 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 5... 2024-03-11T22:47:06.498 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1579873606' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:47:06.499 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:06.317+0000 7f57d4542200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:47:06.499 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:06.429+0000 7f57d4542200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:06.809 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:06.522+0000 7f57d4542200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:47:06.809 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:06.621+0000 7f57d4542200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:47:06.809 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:06 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:06.801+0000 7f57d4542200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:47:08.072 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:07 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:07.786+0000 7f57d4542200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:47:08.072 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:07 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:07.919+0000 7f57d4542200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:47:08.553 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:08.254+0000 7f57d4542200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:47:08.553 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:08.346+0000 7f57d4542200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:47:08.553 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:08.545+0000 7f57d4542200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:08.822 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:08.733+0000 7f57d4542200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:47:09.289 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:09 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:09.021+0000 7f57d4542200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:09.289 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:09 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:09.281+0000 7f57d4542200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:09.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:09 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:09.528+0000 7f57d4542200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:47:10.822 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:10 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:10.366+0000 7f57d4542200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:47:11.277 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:10 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:10.948+0000 7f57d4542200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:47:11.277 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:11.048+0000 7f57d4542200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:47:11.278 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:11.269+0000 7f57d4542200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:47:11.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: Active manager daemon y restarted 2024-03-11T22:47:11.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: Activating manager daemon y 2024-03-11T22:47:11.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T22:47:11.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: mgrmap e6: y(active, starting, since 0.00329245s) 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: Manager daemon y is now available 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:11.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:11 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:12.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: Found migration_current of "None". Setting to last migration. 2024-03-11T22:47:12.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:12.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:47:12.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:47:12.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:12.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:12.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:47:12.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: mgrmap e7: y(active, since 1.00535s) 2024-03-11T22:47:12.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:12 smithi052 ceph-mon[25824]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:47:12.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:47:12.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T22:47:12.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:47:12.849 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:47:12.849 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 5 is available 2024-03-11T22:47:12.850 INFO:teuthology.orchestra.run.smithi052.stdout:Setting orchestrator backend to cephadm... 2024-03-11T22:47:13.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:13 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:13] ENGINE Bus STARTING 2024-03-11T22:47:13.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:13 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:13] ENGINE Serving on http://172.21.15.52:8765 2024-03-11T22:47:14.430 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: mgrmap e8: y(active, since 2s) 2024-03-11T22:47:14.431 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:13] ENGINE Serving on https://172.21.15.52:7150 2024-03-11T22:47:14.431 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:13] ENGINE Bus STARTED 2024-03-11T22:47:14.431 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:14.431 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:14.431 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:14.431 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:14 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:15.385 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T22:47:15.385 INFO:teuthology.orchestra.run.smithi052.stdout:Generating ssh key... 2024-03-11T22:47:15.507 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:15 smithi052 ceph-mon[25824]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-mon[25824]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-mon[25824]: Generating ssh key... 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: Generating public/private rsa key pair. 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: Your identification has been saved in /tmp/tmp6i7fx72w/key. 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: Your public key has been saved in /tmp/tmp6i7fx72w/key.pub. 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: The key fingerprint is: 2024-03-11T22:47:16.641 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: SHA256:M1L/0dW7RfuRv3XE3qzZ8c1GgMLHAWzJ5pLMmvRzKEo ceph-09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: The key's randomart image is: 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: +---[RSA 3072]----+ 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | o.o | 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | * . .| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | o.* . o +| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | ..=.+ +..++| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | ..+So.o. .*+| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | E +.+o.. . +O| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | . . . o . +O| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | . =O| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: | oo=| 2024-03-11T22:47:16.642 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:16 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: +----[SHA256]-----+ 2024-03-11T22:47:18.238 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:47:18.239 INFO:teuthology.orchestra.run.smithi052.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:47:18.239 INFO:teuthology.orchestra.run.smithi052.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T22:47:18.239 INFO:teuthology.orchestra.run.smithi052.stdout:Adding host smithi052... 2024-03-11T22:47:18.494 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:18 smithi052 ceph-mon[25824]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:19.480 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:19 smithi052 ceph-mon[25824]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi052", "addr": "172.21.15.52", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:20.540 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:20 smithi052 ceph-mon[25824]: Deploying cephadm binary to smithi052 2024-03-11T22:47:23.937 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Added host 'smithi052' with addr '172.21.15.52' 2024-03-11T22:47:23.937 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mon service... 2024-03-11T22:47:24.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:24 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:24.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:24 smithi052 ceph-mon[25824]: Added host smithi052 2024-03-11T22:47:24.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:24 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:25.192 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T22:47:25.192 INFO:teuthology.orchestra.run.smithi052.stdout:Deploying unmanaged mgr service... 2024-03-11T22:47:25.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:25 smithi052 ceph-mon[25824]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:25.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:25 smithi052 ceph-mon[25824]: Saving service mon spec with placement count:5 2024-03-11T22:47:25.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:25 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:26.609 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T22:47:27.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:26 smithi052 ceph-mon[25824]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:27.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:26 smithi052 ceph-mon[25824]: Saving service mgr spec with placement count:2 2024-03-11T22:47:27.031 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:26 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:28.678 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:28 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3321232944' entity='client.admin' 2024-03-11T22:47:28.678 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:28 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:28.678 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:28 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:30.120 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling the dashboard module... 2024-03-11T22:47:30.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/4058423082' entity='client.admin' 2024-03-11T22:47:30.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:30.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:30.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:47:30.218 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' 2024-03-11T22:47:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-11T22:47:30.219 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:29 smithi052 ceph-mon[25824]: from='mgr.14120 172.21.15.52:0/2183432360' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]': finished 2024-03-11T22:47:31.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:31 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2295989719' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T22:47:31.322 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:31 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ignoring --setuser ceph since I am not root 2024-03-11T22:47:31.322 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:31 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ignoring --setgroup ceph since I am not root 2024-03-11T22:47:31.844 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:31 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:31.582+0000 7fa12acc2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:47:31.844 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:31 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:31.687+0000 7fa12acc2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:47:32.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2295989719' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T22:47:32.101 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-mon[25824]: mgrmap e9: y(active, since 19s) 2024-03-11T22:47:32.695 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:32.359+0000 7fa12acc2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:47:32.695 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:32.457+0000 7fa12acc2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:47:32.695 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:32.602+0000 7fa12acc2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:47:33.009 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:32.701+0000 7fa12acc2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:47:33.009 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:32 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:32.787+0000 7fa12acc2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:47:33.066 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for the mgr to restart... 2024-03-11T22:47:33.067 INFO:teuthology.orchestra.run.smithi052.stdout:Waiting for mgr epoch 9... 2024-03-11T22:47:33.268 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:33 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/10647928' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:47:33.268 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:33 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:33.060+0000 7fa12acc2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:47:33.268 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:33 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:33.177+0000 7fa12acc2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:33.557 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:33 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:33.269+0000 7fa12acc2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:47:33.557 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:33 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:33.369+0000 7fa12acc2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:47:33.557 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:33 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:33.550+0000 7fa12acc2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:47:34.822 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:34 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:34.540+0000 7fa12acc2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:47:34.823 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:34 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:34.674+0000 7fa12acc2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:47:35.311 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:35 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:35.011+0000 7fa12acc2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:47:35.311 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:35 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:35.103+0000 7fa12acc2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:47:35.311 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:35 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:35.304+0000 7fa12acc2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:35.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:35 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:35.493+0000 7fa12acc2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:47:36.053 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:35 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:35.785+0000 7fa12acc2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:47:36.053 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:36 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:36.046+0000 7fa12acc2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:47:36.322 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:36 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:36.289+0000 7fa12acc2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:47:37.572 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:37 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:37.115+0000 7fa12acc2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:47:38.027 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:37 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:37.697+0000 7fa12acc2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:47:38.027 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:37 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:37.798+0000 7fa12acc2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:47:38.027 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:47:38.020+0000 7fa12acc2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:47:38.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: Active manager daemon y restarted 2024-03-11T22:47:38.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: Activating manager daemon y 2024-03-11T22:47:38.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T22:47:38.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: mgrmap e10: y(active, starting, since 0.00328163s) 2024-03-11T22:47:38.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: Manager daemon y is now available 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:38.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:38 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:39.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:47:39.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:47:39.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:38] ENGINE Bus STARTING 2024-03-11T22:47:39.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:38] ENGINE Serving on https://172.21.15.52:7150 2024-03-11T22:47:39.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:38] ENGINE Serving on http://172.21.15.52:8765 2024-03-11T22:47:39.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: [11/Mar/2024:22:47:38] ENGINE Bus STARTED 2024-03-11T22:47:39.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:47:39.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: mgrmap e11: y(active, since 1.00595s) 2024-03-11T22:47:39.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:39 smithi052 ceph-mon[25824]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:47:39.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:47:39.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T22:47:39.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:47:39.622 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:47:39.622 INFO:teuthology.orchestra.run.smithi052.stdout:mgr epoch 9 is available 2024-03-11T22:47:39.622 INFO:teuthology.orchestra.run.smithi052.stdout:Generating a dashboard self-signed certificate... 2024-03-11T22:47:41.022 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T22:47:41.022 INFO:teuthology.orchestra.run.smithi052.stdout:Creating initial admin user... 2024-03-11T22:47:41.282 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:41 smithi052 ceph-mon[25824]: mgrmap e12: y(active, since 2s) 2024-03-11T22:47:41.283 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:41 smithi052 ceph-mon[25824]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:41.283 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:41 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:41.283 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:41 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: Deploying daemon agent.smithi052 on smithi052 2024-03-11T22:47:42.537 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:42 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:42.544 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nEyWc/2gUFdBsh7NWbA8keZp1zVtMxDdVC08nsVmgbXkYm0JHp18W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710197261, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T22:47:42.544 INFO:teuthology.orchestra.run.smithi052.stdout:Fetching dashboard port number... 2024-03-11T22:47:43.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:43 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3597850870' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T22:47:43.841 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T22:47:43.851 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:47:43.852 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout disabled 2024-03-11T22:47:43.858 INFO:teuthology.orchestra.run.smithi052.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:47:43.858 INFO:teuthology.orchestra.run.smithi052.stdout:systemctl: stdout inactive 2024-03-11T22:47:43.859 INFO:teuthology.orchestra.run.smithi052.stdout:firewalld.service is not enabled 2024-03-11T22:47:43.859 INFO:teuthology.orchestra.run.smithi052.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T22:47:43.861 INFO:teuthology.orchestra.run.smithi052.stdout:Ceph Dashboard is now available at: 2024-03-11T22:47:43.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:43.861 INFO:teuthology.orchestra.run.smithi052.stdout: URL: https://smithi052.front.sepia.ceph.com:8443/ 2024-03-11T22:47:43.861 INFO:teuthology.orchestra.run.smithi052.stdout: User: admin 2024-03-11T22:47:43.861 INFO:teuthology.orchestra.run.smithi052.stdout: Password: e3g9mkxywn 2024-03-11T22:47:43.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:43.862 INFO:teuthology.orchestra.run.smithi052.stdout:Saving cluster configuration to /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config directory 2024-03-11T22:47:43.862 INFO:teuthology.orchestra.run.smithi052.stdout:Enabling autotune for osd_memory_target 2024-03-11T22:47:45.422 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:45 smithi052 ceph-mon[25824]: mgrmap e13: y(active, since 6s) 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T22:47:46.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout: ceph telemetry on 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout:For more information see: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:47:46.612 INFO:teuthology.orchestra.run.smithi052.stdout:Bootstrap complete. 2024-03-11T22:47:46.657 INFO:tasks.cephadm:Fetching config... 2024-03-11T22:47:46.657 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:47:46.657 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T22:47:46.672 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T22:47:46.672 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:47:46.673 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T22:47:46.728 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T22:47:46.728 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:47:46.728 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T22:47:46.870 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T22:47:46.871 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:47:46.871 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T22:47:46.925 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T22:47:46.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:47:47.082 INFO:teuthology.orchestra.run.smithi052.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:47:47.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:47:47.142 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:47:47.150 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:46 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/423099665' entity='client.admin' 2024-03-11T22:47:47.157 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:47:47.203 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8rQirfqATQ+sCPyJh0RiXQc7eQT2/1yfRWO2Cntz0XKTXEaRC1Rj86Fsogjq8okYf0tA3vt2OI15hkgULEqSN5PRu59fMk+MlWeNdtzf05bBiPqXcu3ZfKsFeg6JQEw3WoL+gDgATPpErJ2yqyjdPrb4J1Yx6bPZCHsf9dSmDrRpUpXVAfqNE6sVsoqwVnRgissrG1FclQ9sqMKKtYNF7AhaJImO/iOSG/0GQcaW4vNtBDl2x0NoVQWgLTqaC+1JceoDFF14m8upB7htKGK+Aqgo1t070J6y0nH4NIQWkyGGwHPBicgW0/mXrtFkwRlsEOcSt73G0kzDzjy3JjvUk4YxX3TILipN2zxUDYIAWOabAU1jE5S9r+oap6AKew2DpF0wYra9w99uDjiUXzR0teWWuXH9yqH4oZQRUNJbeu9ok8PuBRQhNPz/YFScL+pgfbuQDAOQoiJODipH6Ws1QWX6V2eu9mlwmUohEcGPOEEJeIEf0SMrEl3IfZKcWJ0= ceph-09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:47:47.218 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T22:47:47.634 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:47:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:49.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:50.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:50 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3526476316' entity='client.admin' 2024-03-11T22:47:50.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:50 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:50.727 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:50 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:50.753 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T22:47:50.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T22:47:51.551 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:47:53.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:53 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:53.762 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:53 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.852 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:54.852 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.852 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:47:54.852 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.853 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:47:54.880 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi116 2024-03-11T22:47:54.880 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:47:54.880 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:47:54.899 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:47:54.899 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:47:54.955 INFO:tasks.cephadm:Adding host smithi116 to orchestrator... 2024-03-11T22:47:54.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch host add smithi116 2024-03-11T22:47:55.194 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:47:55.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:55 smithi052 ceph-mon[25824]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:47:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:57 smithi052 ceph-mon[25824]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:47:59.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:59 smithi052 ceph-mon[25824]: Deploying cephadm binary to smithi116 2024-03-11T22:47:59.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:47:59 smithi052 ceph-mon[25824]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:01.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:01 smithi052 ceph-mon[25824]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:01.472 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi116' with addr '172.21.15.116' 2024-03-11T22:48:02.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:48:02.647 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:02.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:02 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:02.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:02 smithi052 ceph-mon[25824]: Added host smithi116 2024-03-11T22:48:02.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:02 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:02.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:02 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:02.772 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:02 smithi052 ceph-mon[25824]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:03.794 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi116", "caps": []}]: dispatch 2024-03-11T22:48:03.795 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:03 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi116", "caps": []}]': finished 2024-03-11T22:48:04.513 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:48:04.513 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}] 2024-03-11T22:48:04.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:04 smithi052 ceph-mon[25824]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:48:04.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:04 smithi052 ceph-mon[25824]: Deploying daemon agent.smithi116 on smithi116 2024-03-11T22:48:04.816 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:04 smithi052 ceph-mon[25824]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:05.118 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-03-11T22:48:05.118 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:48:05.118 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:48:05.137 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:48:05.137 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:05.194 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-03-11T22:48:05.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch host add smithi152 2024-03-11T22:48:05.439 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:05.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:48:05.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:05.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:05.802 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:05.803 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:05 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:06.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:06 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:06.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:06 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:06.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:06 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:06.771 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:06 smithi052 ceph-mon[25824]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:08.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:08 smithi052 ceph-mon[25824]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:08.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:08 smithi052 ceph-mon[25824]: Deploying cephadm binary to smithi152 2024-03-11T22:48:08.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:08 smithi052 ceph-mon[25824]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:08.379 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:08 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:09.750 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:09.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:09.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:09.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:09.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:10.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:10 smithi052 ceph-mon[25824]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:11.764 INFO:teuthology.orchestra.run.smithi052.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-03-11T22:48:12.450 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:48:12.698 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:13.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:12 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:13.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:12 smithi052 ceph-mon[25824]: Added host smithi152 2024-03-11T22:48:13.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:12 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:13.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:12 smithi052 ceph-mon[25824]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:13.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:12 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:48:13.993 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:13.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:13.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]: dispatch 2024-03-11T22:48:13.994 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]': finished 2024-03-11T22:48:14.552 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:48:14.552 INFO:teuthology.orchestra.run.smithi052.stdout:[{"addr": "172.21.15.52", "hostname": "smithi052", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-03-11T22:48:15.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:14 smithi052 ceph-mon[25824]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:15.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:14 smithi052 ceph-mon[25824]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:48:15.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:14 smithi052 ceph-mon[25824]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:15.195 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T22:48:15.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T22:48:15.435 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: Deploying daemon agent.smithi152 on smithi152 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:16.308 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:16.309 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:17.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:17 smithi052 ceph-mon[25824]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:18.035 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-03-11T22:48:18.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:18 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1614680242' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:48:18.641 INFO:tasks.cephadm:Adding mon.a on smithi052 2024-03-11T22:48:18.641 INFO:tasks.cephadm:Adding mon.b on smithi116 2024-03-11T22:48:18.641 INFO:tasks.cephadm:Adding mon.c on smithi152 2024-03-11T22:48:18.641 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c' 2024-03-11T22:48:18.887 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:48:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:19 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1614680242' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:48:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:19 smithi052 ceph-mon[25824]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:19 smithi052 ceph-mon[25824]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:19 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:19 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:19 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:20.477 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2024-03-11T22:48:20.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:20 smithi052 ceph-mon[25824]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:21.069 DEBUG:teuthology.orchestra.run.smithi116:mon.b> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T22:48:21.072 DEBUG:teuthology.orchestra.run.smithi152:mon.c> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T22:48:21.074 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:48:21.075 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:48:21.795 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:48:21.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c;count:3 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:21.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:21 smithi052 ceph-mon[25824]: Deploying daemon mon.c on smithi152 2024-03-11T22:48:22.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:22 smithi052 ceph-mon[25824]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:24.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:24 smithi052 ceph-mon[25824]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.227 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 podman[27741]: 2024-03-11 22:48:25.221281038 +0000 UTC m=+0.140521862 container create 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T22:48:25.357 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:48:25.357 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"09234c40-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:46:40.844057Z","created":"2024-03-11T22:46:40.844057Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:48:25.359 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-11T22:48:25.477 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 podman[27741]: 2024-03-11 22:48:25.442181889 +0000 UTC m=+0.361422724 container init 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T22:48:25.478 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 podman[27741]: 2024-03-11 22:48:25.448024045 +0000 UTC m=+0.367264872 container start 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:48:25.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:25 smithi052 ceph-mon[25824]: from='client.? 172.21.15.152:0/2092618489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pidfile_write: ignore empty --pid-file 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: load: jerasure load: lrc 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Git sha 0 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:48:25.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: DB SUMMARY 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: DB Session ID: FIUC27NMX3JDTTZPA1VE 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.env: 0x55b95e34dae0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.info_log: 0x55b960add8c0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.statistics: (nil) 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.use_fsync: 0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:48:25.833 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.db_log_dir: 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.wal_dir: 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:48:25.834 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.write_buffer_manager: 0x55b960ab1400 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.unordered_write: 0 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:48:25.835 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.row_cache: None 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.wal_filter: None 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.wal_compression: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:48:25.836 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:48:25.837 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:48:25.837 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:48:25.837 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:48:25.837 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:48:25.838 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:48:25.838 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:48:25.838 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_open_files: -1 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:48:25.839 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Compression algorithms supported: 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kZSTD supported: 0 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.merge_operator: 2024-03-11T22:48:25.840 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_filter: None 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b960add9c0) 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: index_type: 0 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: data_block_index_type: 0 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: index_shortening: 1 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:48:25.841 INFO:journalctl@ceph.mon.c.smithi152.stdout: checksum: 4 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: no_block_cache: 0 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache: 0x55b960ae1090 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache_options: 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: capacity : 536870912 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: num_shard_bits : 4 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: strict_capacity_limit : 0 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache_compressed: (nil) 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: persistent_cache: (nil) 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_size: 4096 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_size_deviation: 10 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_restart_interval: 16 2024-03-11T22:48:25.842 INFO:journalctl@ceph.mon.c.smithi152.stdout: index_block_restart_interval: 1 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: metadata_block_size: 4096 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: partition_filters: 0 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: use_delta_encoding: 1 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: filter_policy: bloomfilter 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: whole_key_filtering: 1 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: verify_compression: 0 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: format_version: 5 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: enable_index_compression: 1 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_align: 0 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: max_auto_readahead_size: 262144 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: prepopulate_block_cache: 0 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:48:25.843 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:48:25.844 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:48:25.845 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression: NoCompression 2024-03-11T22:48:25.845 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:48:25.845 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:48:25.845 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:48:25.845 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.num_levels: 7 2024-03-11T22:48:25.845 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:48:25.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:48:25.847 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:48:25.848 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:48:25.849 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.ttl: 2592000 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:48:25.850 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c96096a0-e2c2-4c97-904f-cd72e2ce0707 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197305478201, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197305478604, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710197305, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c96096a0-e2c2-4c97-904f-cd72e2ce0707", "db_session_id": "FIUC27NMX3JDTTZPA1VE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:48:25.851 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197305478683, "job": 1, "event": "recovery_finished"} 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b960bd2000 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: DB pointer 0x55b960bbc000 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** DB Stats ** 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-11T22:48:25.852 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:48:25.853 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: Block cache BinnedLRUCache@0x55b960ae1090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: using public_addr v2:172.21.15.152:0/0 -> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: starting mon.c rank -1 at public addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] at bind addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:48:25.854 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(???) e0 preinit fsid 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout: e1 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout: legacy client fscid: -1 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout: No filesystems configured 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:48:25.855 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Deploying daemon agent.smithi116 on smithi116 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.856 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Deploying cephadm binary to smithi152 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.857 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.858 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Added host smithi152 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.859 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]: dispatch 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]': finished 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Deploying daemon agent.smithi152 on smithi152 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.860 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/1614680242' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/1614680242' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.861 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c;count:3 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:25.862 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:25.863 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: Deploying daemon mon.c on smithi152 2024-03-11T22:48:25.863 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.863 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:25.863 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: from='client.? 172.21.15.152:0/2092618489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:25.863 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:25 smithi152 ceph-mon[27791]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T22:48:26.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:26 smithi152 bash[27741]: 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 2024-03-11T22:48:26.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:26 smithi152 systemd[1]: Started Ceph mon.c for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:48:27.021 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:48:27.022 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:48:27.257 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:48:31.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: mon.a calling monitor election 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: mon.c calling monitor election 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: monmap e2: 2 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:31.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: fsmap 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: mgrmap e13: y(active, since 52s) 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: overall HEALTH_OK 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:31.075 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:31.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:48:31.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: mon.a calling monitor election 2024-03-11T22:48:31.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: mon.c calling monitor election 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: monmap e2: 2 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: fsmap 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: mgrmap e13: y(active, since 52s) 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: overall HEALTH_OK 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:31.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:32.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:48:32.033 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":2,"fsid":"09234c40-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:48:25.505337Z","created":"2024-03-11T22:46:40.844057Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:48:32.033 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:31 smithi152 ceph-mon[27791]: Deploying daemon mon.b on smithi116 2024-03-11T22:48:32.033 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:31 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:32.035 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 2 2024-03-11T22:48:32.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:31 smithi052 ceph-mon[25824]: Deploying daemon mon.b on smithi116 2024-03-11T22:48:32.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:31 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:33.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:32 smithi052 ceph-mon[25824]: from='client.? 172.21.15.152:0/3105609814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:33.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:32 smithi052 ceph-mon[25824]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:33.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:32 smithi152 ceph-mon[27791]: from='client.? 172.21.15.152:0/3105609814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:33.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:32 smithi152 ceph-mon[27791]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:33.748 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:48:33.748 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:48:33.990 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:48:34.389 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:34 smithi152 ceph-mon[27791]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:34 smithi052 ceph-mon[25824]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:48:34.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Deploying daemon agent.smithi116 on smithi116 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.686 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.687 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:34.687 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.687 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.687 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Deploying cephadm binary to smithi152 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Added host smithi152 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]: dispatch 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]': finished 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Deploying daemon agent.smithi152 on smithi152 2024-03-11T22:48:34.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:34.992 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/1614680242' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/1614680242' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:34.993 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Saving service mon spec with placement smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c;count:3 2024-03-11T22:48:34.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.994 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Deploying daemon mon.c on smithi152 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.? 172.21.15.152:0/2092618489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.a calling monitor election 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.995 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.c calling monitor election 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: monmap e2: 2 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: fsmap 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mgrmap e13: y(active, since 52s) 2024-03-11T22:48:34.996 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: overall HEALTH_OK 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: Deploying daemon mon.b on smithi116 2024-03-11T22:48:34.997 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: from='client.? 172.21.15.152:0/3105609814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 ceph-mon[27552]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T22:48:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 bash[27515]: 8a1c8d1a55fdb5a1cd919ecc3bd3d3ef0615532b8e5379fa4d1e0a71e36bc469 2024-03-11T22:48:34.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:34 smithi116 systemd[1]: Started Ceph mon.b for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:48:40.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:48:40.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: mon.c calling monitor election 2024-03-11T22:48:40.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: mon.a calling monitor election 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:48:40.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: fsmap 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: mgrmap e13: y(active, since 61s) 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: mon.b (rank 2) addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] is down (out of quorum) 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.074 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:48:40.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: mon.c calling monitor election 2024-03-11T22:48:40.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: mon.a calling monitor election 2024-03-11T22:48:40.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: fsmap 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: mgrmap e13: y(active, since 61s) 2024-03-11T22:48:40.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: mon.b (rank 2) addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] is down (out of quorum) 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.189 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-11T22:48:40.189 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":3,"fsid":"09234c40-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:48:34.683917Z","created":"2024-03-11T22:46:40.844057Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:3300","nonce":0},{"type":"v1","addr":"172.21.15.52:6789","nonce":0}]},"addr":"172.21.15.52:6789/0","public_addr":"172.21.15.52:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:48:40.190 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 3 2024-03-11T22:48:40.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:40.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:40.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='client.? 172.21.15.152:0/2388133909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:40.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:40.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:40.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:40 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:40.897 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T22:48:40.897 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:48:41.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='client.? 172.21.15.152:0/2388133909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:41.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:40 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:41.159 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: mon.c calling monitor election 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: mon.a calling monitor election 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: fsmap 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: mgrmap e13: y(active, since 61s) 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: mon.b (rank 2) addr [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] is down (out of quorum) 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='client.? 172.21.15.152:0/2388133909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:41.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:41 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: mon.b calling monitor election 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: mon.b calling monitor election 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: mon.c calling monitor election 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: mon.a calling monitor election 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:42.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: fsmap 2024-03-11T22:48:42.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:42.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: mgrmap e13: y(active, since 63s) 2024-03-11T22:48:42.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:48:42.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: Cluster is now healthy 2024-03-11T22:48:42.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: overall HEALTH_OK 2024-03-11T22:48:42.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:42 smithi052 ceph-mon[25824]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:42.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: mon.b calling monitor election 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: mon.b calling monitor election 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: mon.c calling monitor election 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: mon.a calling monitor election 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: fsmap 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: mgrmap e13: y(active, since 63s) 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: Cluster is now healthy 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: overall HEALTH_OK 2024-03-11T22:48:42.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:42 smithi152 ceph-mon[27791]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: mon.b calling monitor election 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: mon.b calling monitor election 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: mon.c calling monitor election 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: mon.a calling monitor election 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: monmap e3: 3 mons at {a=[v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0],b=[v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: fsmap 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: mgrmap e13: y(active, since 63s) 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: Cluster is now healthy 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: overall HEALTH_OK 2024-03-11T22:48:42.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:42 smithi116 ceph-mon[27552]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.823 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.945 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:43.996 INFO:teuthology.orchestra.run.smithi052.stdout:# minimal ceph.conf for 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:48:43.996 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-03-11T22:48:43.996 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:48:43.996 INFO:teuthology.orchestra.run.smithi052.stdout: mon_host = [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-11T22:48:44.631 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T22:48:44.631 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:48:44.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:48:44.663 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:48:44.663 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:44.732 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:48:44.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:48:44.764 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:48:44.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:44.765 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:48:44.766 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: Reconfiguring daemon mon.b on smithi116 2024-03-11T22:48:44.766 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:44.766 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:44.766 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:44.766 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/3057743276' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:44.766 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:44 smithi116 ceph-mon[27552]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:44.795 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:48:44.795 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:48:44.832 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:48:44.832 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:48:44.903 INFO:tasks.cephadm:Adding mgr.y on smithi052 2024-03-11T22:48:44.903 INFO:tasks.cephadm:Adding mgr.x on smithi116 2024-03-11T22:48:44.904 INFO:tasks.cephadm:Adding mgr.z on smithi152 2024-03-11T22:48:44.904 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi052=y;smithi116=x;smithi152=z' 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: Reconfiguring daemon mon.b on smithi116 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/3057743276' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:44.968 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:44 smithi152 ceph-mon[27791]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: Reconfiguring daemon mon.b on smithi116 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3057743276' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:45.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:44 smithi052 ceph-mon[25824]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:45.183 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:48:46.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:46.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:46.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:48:46.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:46.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:46.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:46.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:45 smithi052 ceph-mon[25824]: Reconfiguring daemon mon.c on smithi152 2024-03-11T22:48:46.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:46.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:46.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:48:46.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:46.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:46.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:46.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:45 smithi152 ceph-mon[27791]: Reconfiguring daemon mon.c on smithi152 2024-03-11T22:48:46.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:46.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:46.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:48:46.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:48:46.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:48:46.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:46.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:45 smithi116 ceph-mon[27552]: Reconfiguring daemon mon.c on smithi152 2024-03-11T22:48:46.910 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:46 smithi052 ceph-mon[25824]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:47.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:46 smithi152 ceph-mon[27791]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:47.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:46 smithi116 ceph-mon[27552]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:47.954 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mgr update... 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=y;smithi116=x;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:48.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: Saving service mgr spec with placement smithi052=y;smithi116=x;smithi152=z;count:3 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: Deploying daemon mgr.z on smithi152 2024-03-11T22:48:48.323 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:48 smithi052 ceph-mon[25824]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:48.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=y;smithi116=x;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: Saving service mgr spec with placement smithi052=y;smithi116=x;smithi152=z;count:3 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.489 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:48.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:48:48.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:48.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: Deploying daemon mgr.z on smithi152 2024-03-11T22:48:48.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:48 smithi116 ceph-mon[27552]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.525 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi052=y;smithi116=x;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: Saving service mgr spec with placement smithi052=y;smithi116=x;smithi152=z;count:3 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: Deploying daemon mgr.z on smithi152 2024-03-11T22:48:48.526 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:48 smithi152 ceph-mon[27791]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:48.565 DEBUG:teuthology.orchestra.run.smithi116:mgr.x> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T22:48:48.568 DEBUG:teuthology.orchestra.run.smithi152:mgr.z> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T22:48:48.608 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T22:48:48.608 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:48:48.608 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:48:48.623 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:48:48.624 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme1n1 2024-03-11T22:48:48.679 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme1n1 2024-03-11T22:48:48.679 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:48.679 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T22:48:48.679 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:48.680 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:48.680 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:48:48.601699196 +0000 2024-03-11T22:48:48.680 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:45:14.746473045 +0000 2024-03-11T22:48:48.680 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:45:14.746473045 +0000 2024-03-11T22:48:48.680 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:45:14.737473155 +0000 2024-03-11T22:48:48.680 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:48:48.746 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:48:48.746 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:48:48.747 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000121378 s, 4.2 MB/s 2024-03-11T22:48:48.748 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:48:48.803 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme2n1 2024-03-11T22:48:48.858 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme2n1 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:48:48.610699083 +0000 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:45:14.896471203 +0000 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:45:14.896471203 +0000 2024-03-11T22:48:48.859 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:45:14.888471301 +0000 2024-03-11T22:48:48.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:48:48.924 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:48:48.924 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:48:48.924 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.9584e-05 s, 5.1 MB/s 2024-03-11T22:48:48.925 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:48:48.981 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme3n1 2024-03-11T22:48:49.036 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme3n1 2024-03-11T22:48:49.036 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:49.036 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:48:49.036 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:49.037 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:49.037 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:48:48.619698970 +0000 2024-03-11T22:48:49.037 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:45:15.045456973 +0000 2024-03-11T22:48:49.037 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:45:15.045456973 +0000 2024-03-11T22:48:49.037 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:45:15.038458092 +0000 2024-03-11T22:48:49.037 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:48:49.102 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:48:49.102 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:48:49.102 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000102656 s, 5.0 MB/s 2024-03-11T22:48:49.104 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:48:49.160 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/nvme4n1 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/nvme4n1 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-11 22:48:48.628698857 +0000 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-11 22:45:15.198432513 +0000 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-11 22:45:15.198432513 +0000 2024-03-11T22:48:49.216 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-11 22:45:15.187434272 +0000 2024-03-11T22:48:49.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:48:49.281 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-11T22:48:49.281 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-11T22:48:49.282 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000118887 s, 4.3 MB/s 2024-03-11T22:48:49.283 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:48:49.339 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:48:49.339 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:48:49.358 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:48:49.358 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/nvme1n1 2024-03-11T22:48:49.416 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/nvme1n1 2024-03-11T22:48:49.416 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:49.416 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T22:48:49.416 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:49.416 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:49.417 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:48:42.552073847 +0000 2024-03-11T22:48:49.417 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:45:16.431252812 +0000 2024-03-11T22:48:49.417 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:45:16.431252812 +0000 2024-03-11T22:48:49.417 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:45:16.418253008 +0000 2024-03-11T22:48:49.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:48:49.487 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:48:49.487 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:48:49.487 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000247006 s, 2.1 MB/s 2024-03-11T22:48:49.488 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:48:49.549 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/nvme2n1 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/nvme2n1 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:48:42.563073671 +0000 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:45:16.577250611 +0000 2024-03-11T22:48:49.607 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:45:16.577250611 +0000 2024-03-11T22:48:49.608 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:45:16.566250779 +0000 2024-03-11T22:48:49.608 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:48:49.676 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:48:49.676 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:48:49.676 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000241635 s, 2.1 MB/s 2024-03-11T22:48:49.678 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:48:49.735 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/nvme3n1 2024-03-11T22:48:49.792 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/nvme3n1 2024-03-11T22:48:49.792 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:49.792 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:48:49.793 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:49.793 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:49.793 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:48:42.575073479 +0000 2024-03-11T22:48:49.793 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:45:16.729248284 +0000 2024-03-11T22:48:49.793 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:45:16.729248284 +0000 2024-03-11T22:48:49.793 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:45:16.721248406 +0000 2024-03-11T22:48:49.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:48:49.861 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:48:49.861 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:48:49.861 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000301461 s, 1.7 MB/s 2024-03-11T22:48:49.863 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:48:49.920 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/nvme4n1 2024-03-11T22:48:49.976 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/nvme4n1 2024-03-11T22:48:49.976 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:49.976 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:48:49.976 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:49.976 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:49.976 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-11 22:48:42.585073319 +0000 2024-03-11T22:48:49.977 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-11 22:45:16.891245804 +0000 2024-03-11T22:48:49.977 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-11 22:45:16.891245804 +0000 2024-03-11T22:48:49.977 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-11 22:45:16.884245911 +0000 2024-03-11T22:48:49.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:48:50.043 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-11T22:48:50.043 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-11T22:48:50.043 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00025525 s, 2.0 MB/s 2024-03-11T22:48:50.044 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:48:50.100 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:48:50.100 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:48:50.116 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:48:50.116 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/nvme1n1 2024-03-11T22:48:50.171 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/nvme1n1 2024-03-11T22:48:50.171 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:50.172 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T22:48:50.172 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:50.172 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:50.172 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:48:42.710029962 +0000 2024-03-11T22:48:50.172 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:45:18.098496542 +0000 2024-03-11T22:48:50.172 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:45:18.098496542 +0000 2024-03-11T22:48:50.173 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:45:18.087496674 +0000 2024-03-11T22:48:50.173 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:48:50.238 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:50 smithi152 ceph-mon[27791]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:50.239 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 systemd[1]: Starting Ceph mgr.z for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:48:50.241 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:48:50.242 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:48:50.242 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000216182 s, 2.4 MB/s 2024-03-11T22:48:50.243 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:48:50.299 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/nvme2n1 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/nvme2n1 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:48:42.720029836 +0000 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:45:18.254494673 +0000 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:45:18.254494673 +0000 2024-03-11T22:48:50.354 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:45:18.247494756 +0000 2024-03-11T22:48:50.355 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:48:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:48:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:48:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000203801 s, 2.5 MB/s 2024-03-11T22:48:50.422 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:48:50.478 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/nvme3n1 2024-03-11T22:48:50.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:50 smithi116 ceph-mon[27552]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:50.531 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 podman[30154]: 2024-03-11 22:48:50.278978341 +0000 UTC m=+0.017770295 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:48:50.531 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 podman[30154]: 2024-03-11 22:48:50.393304643 +0000 UTC m=+0.132096579 container create 6265e2d4f5ac3e7c74b282ea703c473d45d1818faa81a3639c494f58d90c36c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/nvme3n1 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:48:42.731029698 +0000 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:45:18.378493186 +0000 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:45:18.378493186 +0000 2024-03-11T22:48:50.534 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:45:18.366493330 +0000 2024-03-11T22:48:50.535 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:48:50.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:50 smithi052 ceph-mon[25824]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:50.599 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:48:50.599 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:48:50.600 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000200109 s, 2.6 MB/s 2024-03-11T22:48:50.601 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:48:50.658 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/nvme4n1 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/nvme4n1 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-11 22:48:42.742029560 +0000 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-11 22:45:18.525491424 +0000 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-11 22:45:18.525491424 +0000 2024-03-11T22:48:50.713 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-03-11 22:45:18.514491556 +0000 2024-03-11T22:48:50.714 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:48:50.780 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-11T22:48:50.780 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-11T22:48:50.780 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000200128 s, 2.6 MB/s 2024-03-11T22:48:50.781 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:48:50.837 INFO:tasks.cephadm:Deploying osd.0 on smithi052 with /dev/nvme4n1... 2024-03-11T22:48:50.837 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:48:51.076 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:51.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 podman[30154]: 2024-03-11 22:48:50.612578196 +0000 UTC m=+0.351370146 container init 6265e2d4f5ac3e7c74b282ea703c473d45d1818faa81a3639c494f58d90c36c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T22:48:51.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 podman[30154]: 2024-03-11 22:48:50.618046816 +0000 UTC m=+0.356838752 container start 6265e2d4f5ac3e7c74b282ea703c473d45d1818faa81a3639c494f58d90c36c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:48:51.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 ceph-mgr[30297]: -- 172.21.15.152:0/836282059 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570fdb6a1a0 con 0x5570fdb41000 2024-03-11T22:48:51.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 bash[30154]: 6265e2d4f5ac3e7c74b282ea703c473d45d1818faa81a3639c494f58d90c36c6 2024-03-11T22:48:51.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:50 smithi152 systemd[1]: Started Ceph mgr.z for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:48:51.580 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:51.171+0000 7f4fee801200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:48:51.580 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:51.263+0000 7f4fee801200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:48:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:51.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:51.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:48:51.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:51.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:51.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:51 smithi116 ceph-mon[27552]: Deploying daemon mgr.x on smithi116 2024-03-11T22:48:52.004 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:51.905+0000 7f4fee801200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:48:52.004 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:51.997+0000 7f4fee801200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:52.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:51 smithi152 ceph-mon[27791]: Deploying daemon mgr.x on smithi116 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:52.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:51 smithi052 ceph-mon[25824]: Deploying daemon mgr.x on smithi116 2024-03-11T22:48:52.322 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.138+0000 7f4fee801200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:48:52.322 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.233+0000 7f4fee801200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:48:52.322 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.315+0000 7f4fee801200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:48:52.881 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-mon[27791]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:52.882 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.584+0000 7f4fee801200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:48:52.882 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.693+0000 7f4fee801200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:48:52.882 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.780+0000 7f4fee801200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:48:52.882 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:52.875+0000 7f4fee801200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:48:52.886 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:52 smithi116 ceph-mon[27552]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:52.995 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:52 smithi052 ceph-mon[25824]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr: stderr: 10+0 records in 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr:10+0 records out 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0099958 s, 1.0 GB/s 2024-03-11T22:48:52.996 INFO:teuthology.orchestra.run.smithi052.stderr:--> Zapping successful for: 2024-03-11T22:48:53.142 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 podman[29233]: 2024-03-11 22:48:53.128163942 +0000 UTC m=+0.363666294 container init 72598752c6e66dc8d7a89216f38bf12deb920d46aaf09ce33b56fab126842542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T22:48:53.142 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 podman[29233]: 2024-03-11 22:48:53.136264913 +0000 UTC m=+0.371767256 container start 72598752c6e66dc8d7a89216f38bf12deb920d46aaf09ce33b56fab126842542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T22:48:53.330 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:53 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:53.051+0000 7f4fee801200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:48:53.430 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 ceph-mgr[29272]: -- 172.21.15.116:0/3235673229 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1851b01a0 con 0x55f185186c00 2024-03-11T22:48:53.431 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 bash[29233]: 72598752c6e66dc8d7a89216f38bf12deb920d46aaf09ce33b56fab126842542 2024-03-11T22:48:53.431 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 systemd[1]: Started Ceph mgr.x for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:48:53.654 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi052:/dev/nvme4n1 2024-03-11T22:48:53.899 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:48:53.988 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:53.711+0000 7f0bf1b79200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:48:53.988 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:53.819+0000 7f0bf1b79200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:48:54.320 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:54.029+0000 7f4fee801200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:48:54.320 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:54.159+0000 7f4fee801200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:48:54.320 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.320 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.320 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.320 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:54.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:54.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:54.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:48:54.321 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:48:54.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:48:54.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:48:54.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:54.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:54.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:54.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: Reconfiguring daemon mgr.y on smithi052 2024-03-11T22:48:54.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:54 smithi052 ceph-mon[25824]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:54.579 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:54.509+0000 7f4fee801200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:48:54.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:54.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:54.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:54.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: Reconfiguring daemon mgr.y on smithi052 2024-03-11T22:48:54.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-mon[27791]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:54.493+0000 7f0bf1b79200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:54.585+0000 7f0bf1b79200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: Reconfiguring daemon mgr.y on smithi052 2024-03-11T22:48:54.592 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-mon[27552]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:54.918 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:54.727+0000 7f0bf1b79200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:48:54.919 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:54.821+0000 7f0bf1b79200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:48:54.997 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:54.600+0000 7f4fee801200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:48:54.998 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:54.806+0000 7f4fee801200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:48:54.998 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:54.991+0000 7f4fee801200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:48:55.202 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:54 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:54.917+0000 7f0bf1b79200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:48:55.202 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:55.195+0000 7f0bf1b79200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:48:55.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:55.310+0000 7f0bf1b79200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:48:55.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:55.402+0000 7f0bf1b79200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:48:55.566 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:55.283+0000 7f4fee801200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:48:55.566 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:55.559+0000 7f4fee801200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:48:55.830 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:55.802+0000 7f4fee801200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:48:55.840 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:55.506+0000 7f0bf1b79200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:48:55.841 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:55.694+0000 7f0bf1b79200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:48:56.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:56 smithi152 ceph-mon[27791]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:56.449 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:56 smithi052 ceph-mon[25824]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:56.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:56 smithi116 ceph-mon[27552]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:56.751 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:56 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:56.726+0000 7f0bf1b79200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:48:57.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:56 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:56.711+0000 7f4fee801200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:48:57.196 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:56 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:56.862+0000 7f0bf1b79200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:48:57.197 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:57.190+0000 7f0bf1b79200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:57.317+0000 7f4fee801200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:57.416+0000 7f4fee801200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:57.423 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.424 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.478 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:57.279+0000 7f0bf1b79200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:48:57.478 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:57.471+0000 7f0bf1b79200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:48:57.478 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.478 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi052:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:48:57.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:48:57.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:48:57.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:57.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.573 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:57.737 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:57.655+0000 7f0bf1b79200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:48:57.830 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:48:57 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:48:57.646+0000 7f4fee801200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:48:58.199 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:57 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:57.935+0000 7f0bf1b79200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:48:58.199 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:58.192+0000 7f0bf1b79200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:48:58.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:58.425+0000 7f0bf1b79200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:48:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: Standby manager daemon z started 2024-03-11T22:48:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:48:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:58.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: Standby manager daemon z started 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:59.073 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: Standby manager daemon z started 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:48:59.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1843816962' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:48:59.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:48:59.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:48:59.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:48:59.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:48:59.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:48:59.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:59.217+0000 7f0bf1b79200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:48:59.988 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:59.781+0000 7f0bf1b79200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:48:59.988 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:48:59.878+0000 7f0bf1b79200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:48:59.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-mon[27552]: mgrmap e14: y(active, since 80s), standbys: z 2024-03-11T22:48:59.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:48:59.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/3418061130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63880b10-467d-407b-8c32-d986c0a2e54a"}]: dispatch 2024-03-11T22:48:59.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/3418061130' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63880b10-467d-407b-8c32-d986c0a2e54a"}]': finished 2024-03-11T22:48:59.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-mon[27552]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:48:59.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:48:59 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:59 smithi052 ceph-mon[25824]: mgrmap e14: y(active, since 80s), standbys: z 2024-03-11T22:49:00.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:59 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:49:00.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:59 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3418061130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63880b10-467d-407b-8c32-d986c0a2e54a"}]: dispatch 2024-03-11T22:49:00.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:59 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3418061130' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63880b10-467d-407b-8c32-d986c0a2e54a"}]': finished 2024-03-11T22:49:00.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:59 smithi052 ceph-mon[25824]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:49:00.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:48:59 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:00.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:59 smithi152 ceph-mon[27791]: mgrmap e14: y(active, since 80s), standbys: z 2024-03-11T22:49:00.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:59 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:49:00.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:59 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/3418061130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63880b10-467d-407b-8c32-d986c0a2e54a"}]: dispatch 2024-03-11T22:49:00.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:59 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/3418061130' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63880b10-467d-407b-8c32-d986c0a2e54a"}]': finished 2024-03-11T22:49:00.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:59 smithi152 ceph-mon[27791]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:49:00.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:48:59 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:00.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:49:00.091+0000 7f0bf1b79200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:49:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-mon[27552]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-mon[27552]: Standby manager daemon x started 2024-03-11T22:49:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:49:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:49:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:49:00.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:00 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:49:01.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:00 smithi052 ceph-mon[25824]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:01.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:00 smithi052 ceph-mon[25824]: Standby manager daemon x started 2024-03-11T22:49:01.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:00 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:49:01.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:00 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:49:01.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:00 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:49:01.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:00 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:49:01.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:00 smithi152 ceph-mon[27791]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:01.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:00 smithi152 ceph-mon[27791]: Standby manager daemon x started 2024-03-11T22:49:01.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:00 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:49:01.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:00 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:49:01.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:00 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:49:01.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:00 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1442133954' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:49:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:01 smithi116 ceph-mon[27552]: mgrmap e15: y(active, since 82s), standbys: z, x 2024-03-11T22:49:01.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:01 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:49:01.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:01 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2846453678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:02.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:01 smithi052 ceph-mon[25824]: mgrmap e15: y(active, since 82s), standbys: z, x 2024-03-11T22:49:02.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:01 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:49:02.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:01 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2846453678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:02.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:01 smithi152 ceph-mon[27791]: mgrmap e15: y(active, since 82s), standbys: z, x 2024-03-11T22:49:02.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:01 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:49:02.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:01 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2846453678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:02.940 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:02 smithi052 ceph-mon[25824]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:02.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:02 smithi116 ceph-mon[27552]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:03.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:02 smithi152 ceph-mon[27791]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:04.478 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:04 smithi052 ceph-mon[25824]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:04.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:04 smithi116 ceph-mon[27552]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:04.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:04 smithi152 ceph-mon[27791]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:06.238 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:06 smithi052 ceph-mon[25824]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:06.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:06 smithi116 ceph-mon[27552]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:06.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:06 smithi152 ceph-mon[27791]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:07.433 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:49:07.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:07.434 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:07 smithi052 ceph-mon[25824]: Deploying daemon osd.0 on smithi052 2024-03-11T22:49:07.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:49:07.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:07.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:07 smithi116 ceph-mon[27552]: Deploying daemon osd.0 on smithi052 2024-03-11T22:49:07.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:49:07.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:07.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:07 smithi152 ceph-mon[27791]: Deploying daemon osd.0 on smithi052 2024-03-11T22:49:08.380 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:08 smithi052 ceph-mon[25824]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:08.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:08 smithi116 ceph-mon[27552]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:08.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:08 smithi152 ceph-mon[27791]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:10.274 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:10 smithi052 ceph-mon[25824]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:10.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:10 smithi116 ceph-mon[27552]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:10.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:10 smithi152 ceph-mon[27791]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:12.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:11 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:11 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.201 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:11 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:11 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:11 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:11 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:11 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:11 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:11 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:13.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:12 smithi116 ceph-mon[27552]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:13.276 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:12 smithi052 ceph-mon[25824]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:13.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:12 smithi152 ceph-mon[27791]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:14.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:14.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:14.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:14.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.236 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:14 smithi052 ceph-mon[25824]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:14 smithi116 ceph-mon[27552]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:14.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:14.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:14.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:14.582 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:14.582 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:14 smithi152 ceph-mon[27791]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:16.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:16.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:16 smithi052 ceph-mon[25824]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:49:16.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:16.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:16.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:16 smithi116 ceph-mon[27552]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:16 smithi152 ceph-mon[27791]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:49:17.201 INFO:teuthology.orchestra.run.smithi052.stdout:Created osd(s) 0 on host 'smithi052' 2024-03-11T22:49:17.800 DEBUG:teuthology.orchestra.run.smithi052:osd.0> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T22:49:17.803 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/nvme4n1... 2024-03-11T22:49:17.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:49:18.043 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:49:18.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:49:18.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:18.072 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.162 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:49:18.162 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:49:18.162 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-03-11T22:49:18.162 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:18.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]: dispatch 2024-03-11T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:18.331 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:18.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0[38157]: 2024-03-11T22:49:18.281+0000 7f27cc18b700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:49:18.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0[38157]: 2024-03-11T22:49:18.283+0000 7f27c3f76700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: Adjusting osd_memory_target on smithi052 to 16069M 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:19.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:18 smithi116 ceph-mon[27552]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: Adjusting osd_memory_target on smithi052 to 16069M 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:19.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:18 smithi152 ceph-mon[27791]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' 2024-03-11T22:49:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: Adjusting osd_memory_target on smithi052 to 16069M 2024-03-11T22:49:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi052", "root=default"]}]': finished 2024-03-11T22:49:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:49:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:49:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:18 smithi052 ceph-mon[25824]: from='osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271]' entity='osd.0' 2024-03-11T22:49:19.841 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:49:19.841 INFO:teuthology.orchestra.run.smithi116.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:49:19.841 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:49:19.841 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-11T22:49:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-11T22:49:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123581 s, 848 MB/s 2024-03-11T22:49:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-11T22:49:20.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:19 smithi116 ceph-mon[27552]: purged_snaps scrub starts 2024-03-11T22:49:20.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:19 smithi116 ceph-mon[27552]: purged_snaps scrub ok 2024-03-11T22:49:20.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:19 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:20.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:19 smithi116 ceph-mon[27552]: osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271] boot 2024-03-11T22:49:20.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:19 smithi116 ceph-mon[27552]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:49:20.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:19 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:20.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:19 smithi052 ceph-mon[25824]: purged_snaps scrub starts 2024-03-11T22:49:20.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:19 smithi052 ceph-mon[25824]: purged_snaps scrub ok 2024-03-11T22:49:20.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:19 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:20.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:19 smithi052 ceph-mon[25824]: osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271] boot 2024-03-11T22:49:20.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:19 smithi052 ceph-mon[25824]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:49:20.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:19 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:19 smithi152 ceph-mon[27791]: purged_snaps scrub starts 2024-03-11T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:19 smithi152 ceph-mon[27791]: purged_snaps scrub ok 2024-03-11T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:19 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:19 smithi152 ceph-mon[27791]: osd.0 [v2:172.21.15.52:6802/161116271,v1:172.21.15.52:6803/161116271] boot 2024-03-11T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:19 smithi152 ceph-mon[27791]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:19 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:49:20.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi116:/dev/nvme4n1 2024-03-11T22:49:20.679 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:49:21.126 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:20 smithi116 ceph-mon[27552]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:21.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:20 smithi052 ceph-mon[25824]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:21.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:20 smithi152 ceph-mon[27791]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:22.041 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:21 smithi116 ceph-mon[27552]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:49:22.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:21 smithi052 ceph-mon[25824]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:49:22.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:21 smithi152 ceph-mon[27791]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:49:23.309 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:23 smithi116 ceph-mon[27552]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:23.309 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:49:23.310 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:49:23.310 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:23.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:23 smithi052 ceph-mon[25824]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:23.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:49:23.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:49:23.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:23 smithi152 ceph-mon[27791]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:24.306 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:24 smithi116 ceph-mon[27552]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:49:24.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:24 smithi052 ceph-mon[25824]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:49:24.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:24 smithi152 ceph-mon[27791]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:49:25.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:25 smithi052 ceph-mon[25824]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:25.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:25 smithi152 ceph-mon[27791]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:25.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:25 smithi116 ceph-mon[27552]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:26.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:26 smithi052 ceph-mon[25824]: from='client.? 172.21.15.116:0/33759425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]: dispatch 2024-03-11T22:49:26.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:26 smithi052 ceph-mon[25824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]: dispatch 2024-03-11T22:49:26.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:26 smithi052 ceph-mon[25824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]': finished 2024-03-11T22:49:26.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:26 smithi052 ceph-mon[25824]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:49:26.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:26 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:26.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:26 smithi152 ceph-mon[27791]: from='client.? 172.21.15.116:0/33759425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]: dispatch 2024-03-11T22:49:26.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:26 smithi152 ceph-mon[27791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]: dispatch 2024-03-11T22:49:26.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:26 smithi152 ceph-mon[27791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]': finished 2024-03-11T22:49:26.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:26 smithi152 ceph-mon[27791]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:49:26.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:26 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:26.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:26 smithi116 ceph-mon[27552]: from='client.? 172.21.15.116:0/33759425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]: dispatch 2024-03-11T22:49:26.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:26 smithi116 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]: dispatch 2024-03-11T22:49:26.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:26 smithi116 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74ef0290-69e4-4bf0-89d6-92e9a43d5055"}]': finished 2024-03-11T22:49:26.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:26 smithi116 ceph-mon[27552]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:49:26.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:26 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:27.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:27 smithi052 ceph-mon[25824]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:27.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:27 smithi052 ceph-mon[25824]: from='client.? 172.21.15.116:0/1965965880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:27.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:27 smithi152 ceph-mon[27791]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:27.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:27 smithi152 ceph-mon[27791]: from='client.? 172.21.15.116:0/1965965880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:27.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:27 smithi116 ceph-mon[27552]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:27.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:27 smithi116 ceph-mon[27552]: from='client.? 172.21.15.116:0/1965965880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:29 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:29 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:29 smithi116 ceph-mon[27552]: Detected new or changed devices on smithi052 2024-03-11T22:49:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:29 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:29 smithi116 ceph-mon[27552]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:29 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:29 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:29 smithi052 ceph-mon[25824]: Detected new or changed devices on smithi052 2024-03-11T22:49:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:29 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:29 smithi052 ceph-mon[25824]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:29.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:29 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:29 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:29 smithi152 ceph-mon[27791]: Detected new or changed devices on smithi052 2024-03-11T22:49:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:29 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:29 smithi152 ceph-mon[27791]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:30.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:30 smithi116 ceph-mon[27552]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:30.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:30 smithi052 ceph-mon[25824]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:30.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:30 smithi152 ceph-mon[27791]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:32 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:49:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:32 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:32 smithi052 ceph-mon[25824]: Deploying daemon osd.1 on smithi116 2024-03-11T22:49:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:32 smithi052 ceph-mon[25824]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:32.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:32 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:49:32.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:32 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:32.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:32 smithi116 ceph-mon[27552]: Deploying daemon osd.1 on smithi116 2024-03-11T22:49:32.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:32 smithi116 ceph-mon[27552]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:32.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:32 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:49:32.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:32 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:32.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:32 smithi152 ceph-mon[27791]: Deploying daemon osd.1 on smithi116 2024-03-11T22:49:32.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:32 smithi152 ceph-mon[27791]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:34 smithi052 ceph-mon[25824]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:34.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:34 smithi116 ceph-mon[27552]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:34.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:34 smithi152 ceph-mon[27791]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.864 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:36 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.955 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:36.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:36.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:36 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:36 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:38.313 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:38 smithi116 ceph-mon[27552]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:38.452 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:38 smithi152 ceph-mon[27791]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:38.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:38 smithi052 ceph-mon[25824]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.693 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T22:49:39.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:49:39.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:39.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: Reconfiguring daemon osd.1 on smithi116 2024-03-11T22:49:39.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.694 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:39 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: Reconfiguring daemon osd.1 on smithi116 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:39 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:49:39.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:39.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: Reconfiguring daemon osd.1 on smithi116 2024-03-11T22:49:39.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:39.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:39 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:40.685 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:40 smithi116 ceph-mon[27552]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T22:49:40.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:40 smithi052 ceph-mon[25824]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T22:49:40.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:40 smithi152 ceph-mon[27791]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T22:49:41.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:41 smithi052 ceph-mon[25824]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:41.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:41 smithi116 ceph-mon[27552]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:41.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:41 smithi152 ceph-mon[27791]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:42.735 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:42 smithi116 ceph-mon[27552]: from='osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:49:42.735 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:42 smithi116 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:49:42.796 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2024-03-11T22:49:42.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:42 smithi052 ceph-mon[25824]: from='osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:49:42.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:42 smithi052 ceph-mon[25824]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:49:42.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:42 smithi152 ceph-mon[27791]: from='osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:49:42.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:42 smithi152 ceph-mon[27791]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: from='osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:43.607 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:43.639 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T22:49:43.642 INFO:tasks.cephadm:Deploying osd.2 on smithi152 with /dev/nvme4n1... 2024-03-11T22:49:43.642 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:49:43.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:49:43.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:49:43.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: from='osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-11T22:49:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-11T22:49:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:43.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:43 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:43.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:49:43.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:49:43.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:43.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: from='osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-11T22:49:43.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-11T22:49:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:43.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:43 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:43.892 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:49:44.238 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1[33089]: 2024-03-11T22:49:43.949+0000 7fa5b7735700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:49:44.238 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:49:43 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1[33089]: 2024-03-11T22:49:43.951+0000 7fa5af520700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:49:44.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:44 smithi152 ceph-mon[27791]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-11T22:49:44.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:44 smithi152 ceph-mon[27791]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:49:44.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:44 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:44.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:44 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:44.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:44 smithi152 ceph-mon[27791]: from='osd.1 ' entity='osd.1' 2024-03-11T22:49:44.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:44 smithi052 ceph-mon[25824]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-11T22:49:44.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:44 smithi052 ceph-mon[25824]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:49:44.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:44 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:44.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:44 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:44.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:44 smithi052 ceph-mon[25824]: from='osd.1 ' entity='osd.1' 2024-03-11T22:49:44.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:44 smithi116 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-11T22:49:44.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:44 smithi116 ceph-mon[27552]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:49:44.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:44 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:44.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:44 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:44.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:44 smithi116 ceph-mon[27552]: from='osd.1 ' entity='osd.1' 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: purged_snaps scrub starts 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: purged_snaps scrub ok 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: Adjusting osd_memory_target on smithi116 to 16064M 2024-03-11T22:49:45.580 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709] boot 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:49:45.581 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:45 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:45.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: purged_snaps scrub starts 2024-03-11T22:49:45.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: purged_snaps scrub ok 2024-03-11T22:49:45.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: Adjusting osd_memory_target on smithi116 to 16064M 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709] boot 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:49:45.822 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:45 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:45.868 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:49:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:49:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:49:45.869 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-11T22:49:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-11T22:49:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125847 s, 833 MB/s 2024-03-11T22:49:45.869 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: purged_snaps scrub starts 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: purged_snaps scrub ok 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: Adjusting osd_memory_target on smithi116 to 16064M 2024-03-11T22:49:45.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: osd.1 [v2:172.21.15.116:6800/1231550709,v1:172.21.15.116:6801/1231550709] boot 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:49:45.989 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:45 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:49:46.490 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi152:/dev/nvme4n1 2024-03-11T22:49:46.724 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:49:46.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:46 smithi052 ceph-mon[25824]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T22:49:46.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:46 smithi052 ceph-mon[25824]: Cluster is now healthy 2024-03-11T22:49:46.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:46 smithi052 ceph-mon[25824]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:49:46.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:46 smithi152 ceph-mon[27791]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T22:49:46.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:46 smithi152 ceph-mon[27791]: Cluster is now healthy 2024-03-11T22:49:46.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:46 smithi152 ceph-mon[27791]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:49:46.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:46 smithi116 ceph-mon[27552]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T22:49:46.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:46 smithi116 ceph-mon[27552]: Cluster is now healthy 2024-03-11T22:49:46.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:46 smithi116 ceph-mon[27552]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:49:47.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:47 smithi052 ceph-mon[25824]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:47.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:47 smithi152 ceph-mon[27791]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:47.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:47 smithi116 ceph-mon[27552]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:49.398 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.398 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.398 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.398 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:49:49.398 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:49:49.398 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:49:49.399 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:49.399 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:49 smithi152 ceph-mon[27791]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:49.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:49:49.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:49:49.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:49:49.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:49.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:49 smithi116 ceph-mon[27552]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:49.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:49.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:49:49.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:49:49.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:49:49.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:49:49.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:49 smithi052 ceph-mon[25824]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:51.751 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:51 smithi052 ceph-mon[25824]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:51.752 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:51 smithi052 ceph-mon[25824]: from='client.? 172.21.15.152:0/2601974492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]: dispatch 2024-03-11T22:49:51.752 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:51 smithi052 ceph-mon[25824]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]: dispatch 2024-03-11T22:49:51.752 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:51 smithi052 ceph-mon[25824]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]': finished 2024-03-11T22:49:51.752 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:51 smithi052 ceph-mon[25824]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:49:51.752 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:51 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:49:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:51 smithi116 ceph-mon[27552]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:51 smithi116 ceph-mon[27552]: from='client.? 172.21.15.152:0/2601974492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]: dispatch 2024-03-11T22:49:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:51 smithi116 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]: dispatch 2024-03-11T22:49:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:51 smithi116 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]': finished 2024-03-11T22:49:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:51 smithi116 ceph-mon[27552]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:49:51.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:51 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:49:52.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:51 smithi152 ceph-mon[27791]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:52.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:51 smithi152 ceph-mon[27791]: from='client.? 172.21.15.152:0/2601974492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]: dispatch 2024-03-11T22:49:52.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:51 smithi152 ceph-mon[27791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]: dispatch 2024-03-11T22:49:52.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:51 smithi152 ceph-mon[27791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c88b7ec-27d6-4f07-b2ae-aca49f3afa35"}]': finished 2024-03-11T22:49:52.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:51 smithi152 ceph-mon[27791]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:49:52.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:51 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:49:52.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:52 smithi116 ceph-mon[27552]: from='client.? 172.21.15.152:0/1609545917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:53.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:52 smithi052 ceph-mon[25824]: from='client.? 172.21.15.152:0/1609545917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:53.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:52 smithi152 ceph-mon[27791]: from='client.? 172.21.15.152:0/1609545917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:49:53.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:53 smithi116 ceph-mon[27552]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:54.034 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:53 smithi152 ceph-mon[27791]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:54.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:53 smithi052 ceph-mon[25824]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:55.981 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:55 smithi152 ceph-mon[27791]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:55.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:55 smithi116 ceph-mon[27552]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:56.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:55 smithi052 ceph-mon[25824]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:57.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:57 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:57 smithi116 ceph-mon[27552]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:57.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:57 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:57 smithi052 ceph-mon[25824]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:57.572 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.573 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.573 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:57 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:57.573 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:57 smithi152 ceph-mon[27791]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:58.960 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:58.960 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:58.960 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:58 smithi116 ceph-mon[27552]: Detected new or changed devices on smithi116 2024-03-11T22:49:58.960 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:58 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:58.984 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:58.984 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:58.984 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:58 smithi152 ceph-mon[27791]: Detected new or changed devices on smithi116 2024-03-11T22:49:58.984 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:58 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:59.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:59.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:59.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:58 smithi052 ceph-mon[25824]: Detected new or changed devices on smithi116 2024-03-11T22:49:59.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:58 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:49:59.900 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:49:59 smithi152 ceph-mon[27791]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:49:59.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:49:59 smithi116 ceph-mon[27552]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:49:59 smithi052 ceph-mon[25824]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:00.824 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:00 smithi116 ceph-mon[27552]: overall HEALTH_OK 2024-03-11T22:50:00.978 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:00 smithi152 ceph-mon[27791]: overall HEALTH_OK 2024-03-11T22:50:01.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:00 smithi052 ceph-mon[25824]: overall HEALTH_OK 2024-03-11T22:50:01.955 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:01 smithi152 ceph-mon[27791]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:01.955 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:01 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:50:01.955 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:01 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:01.955 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:01 smithi152 ceph-mon[27791]: Deploying daemon osd.2 on smithi152 2024-03-11T22:50:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:01 smithi116 ceph-mon[27552]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:01 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:50:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:01 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:01.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:01 smithi116 ceph-mon[27552]: Deploying daemon osd.2 on smithi152 2024-03-11T22:50:02.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:01 smithi052 ceph-mon[25824]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:02.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:01 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:50:02.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:01 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:02.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:01 smithi052 ceph-mon[25824]: Deploying daemon osd.2 on smithi152 2024-03-11T22:50:03.806 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:03 smithi152 ceph-mon[27791]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:03.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:03 smithi116 ceph-mon[27552]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:04.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:03 smithi052 ceph-mon[25824]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:05.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:05 smithi116 ceph-mon[27552]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:06.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:05 smithi052 ceph-mon[25824]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:06.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:05 smithi152 ceph-mon[27791]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:07 smithi052 ceph-mon[25824]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:07.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:07.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:07.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:07.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:07 smithi152 ceph-mon[27791]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:07.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:07.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:07 smithi116 ceph-mon[27552]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:09.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:09 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.472 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:09 smithi052 ceph-mon[25824]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:09.569 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:09 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.569 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:09 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.569 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:09 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.569 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:09 smithi152 ceph-mon[27791]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:09.617 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 2 on host 'smithi152' 2024-03-11T22:50:09.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:09 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:09 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:09 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:09.738 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:09 smithi116 ceph-mon[27552]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:10.289 DEBUG:teuthology.orchestra.run.smithi152:osd.2> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T22:50:10.291 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-11T22:50:10.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:50:10.760 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:10.881 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: Adjusting osd_memory_target on smithi152 to 16064M 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:50:10.882 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:10 smithi052 ceph-mon[25824]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:50:10.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: Adjusting osd_memory_target on smithi152 to 16064M 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:50:10.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:10 smithi116 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:50:11.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: Adjusting osd_memory_target on smithi152 to 16064M 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:50:11.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:10 smithi152 ceph-mon[27791]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:50:11.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:11 smithi052 ceph-mon[25824]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:50:11.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:11 smithi052 ceph-mon[25824]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:50:11.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:11 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:11.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:11 smithi052 ceph-mon[25824]: from='osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-11T22:50:11.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:11 smithi052 ceph-mon[25824]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-11T22:50:11.781 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:11 smithi052 ceph-mon[25824]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:11.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:11 smithi116 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:50:11.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:11 smithi116 ceph-mon[27552]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:50:11.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:11 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:11.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:11 smithi116 ceph-mon[27552]: from='osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-11T22:50:11.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:11 smithi116 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-11T22:50:11.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:11 smithi116 ceph-mon[27552]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:12.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-mon[27791]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:50:12.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-mon[27791]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:50:12.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:12.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-mon[27791]: from='osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-11T22:50:12.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-mon[27791]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-11T22:50:12.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-mon[27791]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:12.080 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2[34390]: 2024-03-11T22:50:11.637+0000 7fe84d18f700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:50:12.080 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:11 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2[34390]: 2024-03-11T22:50:11.641+0000 7fe848f82700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:50:12.811 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:12 smithi052 ceph-mon[25824]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-11T22:50:12.811 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:12 smithi052 ceph-mon[25824]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:50:12.811 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:12 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:12.811 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:12 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:12.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:12 smithi116 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-11T22:50:12.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:12 smithi116 ceph-mon[27552]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:50:12.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:12 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:12.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:12 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:13.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:12 smithi152 ceph-mon[27791]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-11T22:50:13.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:12 smithi152 ceph-mon[27791]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:50:13.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:12 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:13.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:12 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:13.272 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:50:13.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: purged_snaps scrub starts 2024-03-11T22:50:13.831 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: purged_snaps scrub ok 2024-03-11T22:50:13.832 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155] boot 2024-03-11T22:50:13.832 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:50:13.832 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:13.832 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:13.832 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:13 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2465283385' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:50:13.861 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1710197412,"num_in_osds":3,"osd_in_since":1710197391,"num_remapped_pgs":0} 2024-03-11T22:50:13.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:50:13.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: purged_snaps scrub starts 2024-03-11T22:50:13.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: purged_snaps scrub ok 2024-03-11T22:50:13.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155] boot 2024-03-11T22:50:13.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:50:13.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:13.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:13.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:13 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2465283385' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:50:14.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: purged_snaps scrub starts 2024-03-11T22:50:14.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: purged_snaps scrub ok 2024-03-11T22:50:14.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: osd.2 [v2:172.21.15.152:6800/3140177155,v1:172.21.15.152:6801/3140177155] boot 2024-03-11T22:50:14.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:50:14.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:14.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:50:14.082 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:13 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2465283385' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:50:14.103 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:14.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:14 smithi052 ceph-mon[25824]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:50:14.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:14 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:50:14.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:14 smithi152 ceph-mon[27791]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:50:14.938 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:14 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:50:14.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:14 smithi116 ceph-mon[27552]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:50:14.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:14 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:15.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:15 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:15.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:50:15.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:50:15.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:50:15.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:15.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:15.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:15.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:15 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:15.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:50:15.989 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":21,"fsid":"09234c40-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:46:44.273030+0000","modified":"2024-03-11T22:50:15.650601+0000","last_up_change":"2024-03-11T22:50:12.635056+0000","last_in_change":"2024-03-11T22:49:51.280321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:50:14.378883+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63880b10-467d-407b-8c32-d986c0a2e54a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6803","nonce":161116271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6805","nonce":161116271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6809","nonce":161116271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6807","nonce":161116271}]},"public_addr":"172.21.15.52:6803/161116271","cluster_addr":"172.21.15.52:6805/161116271","heartbeat_back_addr":"172.21.15.52:6809/161116271","heartbeat_front_addr":"172.21.15.52:6807/161116271","state":["exists","up"]},{"osd":1,"uuid":"74ef0290-69e4-4bf0-89d6-92e9a43d5055","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6801","nonce":1231550709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6803","nonce":1231550709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6807","nonce":1231550709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6805","nonce":1231550709}]},"public_addr":"172.21.15.116:6801/1231550709","cluster_addr":"172.21.15.116:6803/1231550709","heartbeat_back_addr":"172.21.15.116:6807/1231550709","heartbeat_front_addr":"172.21.15.116:6805/1231550709","state":["exists","up"]},{"osd":2,"uuid":"0c88b7ec-27d6-4f07-b2ae-aca49f3afa35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6801","nonce":3140177155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6803","nonce":3140177155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6807","nonce":3140177155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6805","nonce":3140177155}]},"public_addr":"172.21.15.152:6801/3140177155","cluster_addr":"172.21.15.152:6803/3140177155","heartbeat_back_addr":"172.21.15.152:6807/3140177155","heartbeat_front_addr":"172.21.15.152:6805/3140177155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:49:17.187764+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:49:43.057891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:50:10.637190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/2870261962":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6801/1990955331":"2024-03-12T22:47:38.033876+0000","172.21.15.52:0/3377302881":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6800/1990955331":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6801/3464055426":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/664002337":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6800/3464055426":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/1304186236":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/1596447173":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/3021375888":"2024-03-12T22:47:11.284060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:50:16.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:50:16.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:50:16.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:50:16.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:16.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:16.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:16.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:15 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:16.281 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40634]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:50:16.281 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40634]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:16.281 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40634]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:16.281 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40634]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:16.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40656]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:50:16.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40656]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:16.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40656]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:16.571 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:50:16 smithi052 sudo[40656]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:16.656 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:50:14.378883+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T22:50:16.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T22:50:16.824 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34705]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:50:16.825 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34705]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:16.825 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34705]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:16.825 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34705]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:16.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:16 smithi116 ceph-mon[27552]: Detected new or changed devices on smithi152 2024-03-11T22:50:16.825 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:16 smithi116 ceph-mon[27552]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:50:16.826 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:50:16.826 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:16 smithi116 ceph-mon[27552]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:50:16.826 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:16 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/591598400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:16.826 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:16 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:16.905 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:17.027 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:16 smithi052 ceph-mon[25824]: Detected new or changed devices on smithi152 2024-03-11T22:50:17.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:16 smithi052 ceph-mon[25824]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:50:17.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:50:17.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:16 smithi052 ceph-mon[25824]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:50:17.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:16 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/591598400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:17.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:16 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:16 smithi152 ceph-mon[27791]: Detected new or changed devices on smithi152 2024-03-11T22:50:17.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:16 smithi152 ceph-mon[27791]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:50:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:50:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:16 smithi152 ceph-mon[27791]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:50:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:16 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/591598400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:17.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:16 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34776]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:50:17.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34776]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:17.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34776]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:17.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:50:16 smithi116 sudo[34776]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:17.344 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35506]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:50:17.345 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35506]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:17.345 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35506]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:17.345 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35506]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:17.715 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35509]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:50:17.715 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35509]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:17.715 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35509]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:17.715 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:50:17 smithi152 sudo[35509]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:17.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40769]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:50:17.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40769]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:17.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40769]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:17.897 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40769]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:17.994 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:17 smithi152 ceph-mon[27791]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:18.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:50:18.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:18.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:17 smithi116 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:18.252 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40780]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40780]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40780]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 sudo[40780]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:18.253 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:17 smithi052 ceph-mon[25824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:18.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:50:18.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:18.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35574]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:18.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35574]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:18.755 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35585]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:50:18.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35585]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:18.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35585]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:18.756 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 sudo[35585]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:18.795 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2024-03-11T22:50:18.963 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34832]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:50:18.963 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34832]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:18.963 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34832]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:18.963 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34832]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:19.070 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:19.070 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:50:19.070 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: Cluster is now healthy 2024-03-11T22:50:19.070 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.070 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.071 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:18 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/3433679386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:50:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34842]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34842]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34842]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 sudo[34842]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: Cluster is now healthy 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:18 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/3433679386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:50:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:50:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: Cluster is now healthy 2024-03-11T22:50:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:19.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:18 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3433679386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:50:19.394 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi052 2024-03-11T22:50:19.395 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi052=foo.a' 2024-03-11T22:50:19.639 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:50:20.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:19 smithi152 ceph-mon[27791]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:20.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:19 smithi152 ceph-mon[27791]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:20.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:19 smithi152 ceph-mon[27791]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:20.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:19 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:20.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:19 smithi116 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:20.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:19 smithi116 ceph-mon[27552]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:20.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:19 smithi116 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:20.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:19 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:20.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:19 smithi052 ceph-mon[25824]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:50:20.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:19 smithi052 ceph-mon[25824]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:20.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:19 smithi052 ceph-mon[25824]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:50:20.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:19 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:22.148 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled rgw.foo.a update... 2024-03-11T22:50:22.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:21 smithi116 ceph-mon[27552]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:22.301 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:21 smithi052 ceph-mon[25824]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:22.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:21 smithi152 ceph-mon[27791]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:22.738 DEBUG:teuthology.orchestra.run.smithi052:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T22:50:22.740 INFO:tasks.cephadm:Adding prometheus.a on smithi116 2024-03-11T22:50:22.740 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi116=a' 2024-03-11T22:50:22.975 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:23.351 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.352 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:50:23.352 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:50:23.352 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.352 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:23.352 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: Deploying daemon rgw.foo.a on smithi052 2024-03-11T22:50:23.352 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:23 smithi152 ceph-mon[27791]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:23.487 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: Deploying daemon rgw.foo.a on smithi052 2024-03-11T22:50:23.488 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:23 smithi116 ceph-mon[27552]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi052=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:50:23.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:50:23.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:23.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:23.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: Deploying daemon rgw.foo.a on smithi052 2024-03-11T22:50:23.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:23 smithi052 ceph-mon[25824]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:24.809 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:50:24 smithi052 podman[41174]: 2024-03-11 22:50:24.661381626 +0000 UTC m=+0.143765096 container create d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:50:24.887 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled prometheus update... 2024-03-11T22:50:25.321 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:50:24 smithi052 podman[41174]: 2024-03-11 22:50:24.916118898 +0000 UTC m=+0.398502379 container init d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T22:50:25.321 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:50:24 smithi052 podman[41174]: 2024-03-11 22:50:24.921638409 +0000 UTC m=+0.404021877 container start d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:50:25.321 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:50:25 smithi052 bash[41174]: d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 2024-03-11T22:50:25.321 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:50:25 smithi052 systemd[1]: Started Ceph rgw.foo.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:50:25.659 DEBUG:teuthology.orchestra.run.smithi116:prometheus.a> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T22:50:25.661 INFO:tasks.cephadm:Adding node-exporter.a on smithi052 2024-03-11T22:50:25.662 INFO:tasks.cephadm:Adding node-exporter.b on smithi116 2024-03-11T22:50:25.662 INFO:tasks.cephadm:Adding node-exporter.c on smithi152 2024-03-11T22:50:25.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi052=a;smithi116=b;smithi152=c' 2024-03-11T22:50:25.899 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:50:25.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:25.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:25.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: Saving service prometheus spec with placement smithi116=a;count:1 2024-03-11T22:50:25.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:25.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:25.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:25 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: Saving service prometheus spec with placement smithi116=a;count:1 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.024 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.025 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.025 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.025 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:26.025 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:26.025 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:25 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:26.137 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:26.137 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:26.137 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: Saving service prometheus spec with placement smithi116=a;count:1 2024-03-11T22:50:26.137 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.137 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.137 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:26.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:26.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:26.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:25 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:26.998 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:26 smithi052 ceph-mon[25824]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-11T22:50:26.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:26 smithi052 ceph-mon[25824]: Checking dashboard <-> RGW credentials 2024-03-11T22:50:26.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:26 smithi052 ceph-mon[25824]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:50:26.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:26 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1816356362' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:50:26.999 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:26 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1722078843' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:50:27.183 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:26 smithi152 ceph-mon[27791]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-11T22:50:27.183 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:26 smithi152 ceph-mon[27791]: Checking dashboard <-> RGW credentials 2024-03-11T22:50:27.183 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:26 smithi152 ceph-mon[27791]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:50:27.183 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:26 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/1816356362' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:50:27.183 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:26 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/1722078843' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:50:27.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:26 smithi116 ceph-mon[27552]: Saving service rgw.foo.a spec with placement smithi052=foo.a;count:1 2024-03-11T22:50:27.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:26 smithi116 ceph-mon[27552]: Checking dashboard <-> RGW credentials 2024-03-11T22:50:27.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:26 smithi116 ceph-mon[27552]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:50:27.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:26 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/1816356362' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:50:27.239 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:26 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/1722078843' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:50:27.651 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled node-exporter update... 2024-03-11T22:50:28.010 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:27 smithi152 ceph-mon[27791]: pgmap v101: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:28.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:27 smithi152 ceph-mon[27791]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:50:28.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:27 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/1816356362' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:50:28.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:27 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/1722078843' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:50:28.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:27 smithi152 ceph-mon[27791]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:50:28.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:27 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:28.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:27 smithi052 ceph-mon[25824]: pgmap v101: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:28.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:27 smithi052 ceph-mon[25824]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:50:28.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:27 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1816356362' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:50:28.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:27 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/1722078843' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:50:28.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:27 smithi052 ceph-mon[25824]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:50:28.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:27 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:28.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:27 smithi116 ceph-mon[27552]: pgmap v101: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:28.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:27 smithi116 ceph-mon[27552]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:50:28.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:27 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/1816356362' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:50:28.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:27 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/1722078843' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:50:28.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:27 smithi116 ceph-mon[27552]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:50:28.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:27 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:28.366 DEBUG:teuthology.orchestra.run.smithi052:node-exporter.a> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T22:50:28.369 DEBUG:teuthology.orchestra.run.smithi116:node-exporter.b> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T22:50:28.372 DEBUG:teuthology.orchestra.run.smithi152:node-exporter.c> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-11T22:50:28.374 INFO:tasks.cephadm:Adding alertmanager.a on smithi052 2024-03-11T22:50:28.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi052=a' 2024-03-11T22:50:28.650 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:50:29.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi052=a;smithi116=b;smithi152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:29.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: Saving service node-exporter spec with placement smithi052=a;smithi116=b;smithi152=c;count:3 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:28 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi052=a;smithi116=b;smithi152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: Saving service node-exporter spec with placement smithi052=a;smithi116=b;smithi152=c;count:3 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.322 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:28 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi052=a;smithi116=b;smithi152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:29.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: Saving service node-exporter spec with placement smithi052=a;smithi116=b;smithi152=c;count:3 2024-03-11T22:50:29.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:50:29.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:29.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:28 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:30.197 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:29 smithi152 ceph-mon[27791]: pgmap v104: 65 pgs: 9 creating+peering, 15 active+clean, 41 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T22:50:30.197 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:29 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:50:30.197 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:29 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:50:30.197 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:29 smithi152 ceph-mon[27791]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:50:30.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:29 smithi116 ceph-mon[27552]: pgmap v104: 65 pgs: 9 creating+peering, 15 active+clean, 41 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T22:50:30.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:29 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:50:30.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:29 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:50:30.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:29 smithi116 ceph-mon[27552]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:50:30.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:29 smithi052 ceph-mon[25824]: pgmap v104: 65 pgs: 9 creating+peering, 15 active+clean, 41 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T22:50:30.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:29 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:50:30.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:29 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:50:30.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:29 smithi052 ceph-mon[25824]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:50:30.522 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled alertmanager update... 2024-03-11T22:50:31.150 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:30 smithi152 ceph-mon[27791]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:50:31.150 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:30 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:50:31.150 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:30 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:50:31.150 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:30 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:31.150 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:30 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:50:31.176 DEBUG:teuthology.orchestra.run.smithi052:alertmanager.a> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T22:50:31.179 INFO:tasks.cephadm:Adding grafana.a on smithi116 2024-03-11T22:50:31.179 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi116=a' 2024-03-11T22:50:31.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:30 smithi052 ceph-mon[25824]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:50:31.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:30 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:50:31.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:30 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:50:31.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:30 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:31.200 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:30 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:50:31.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:30 smithi116 ceph-mon[27552]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:50:31.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:30 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:50:31.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:30 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:50:31.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:30 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:31.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:30 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:50:31.411 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:50:32.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:32.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-03-11T22:50:32.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: pgmap v107: 97 pgs: 29 creating+peering, 56 active+clean, 12 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T22:50:32.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:50:32.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:50:32.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:50:32.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:50:32.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:31 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:50:32.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:32.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-03-11T22:50:32.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: pgmap v107: 97 pgs: 29 creating+peering, 56 active+clean, 12 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T22:50:32.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:50:32.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:50:32.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:50:32.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:50:32.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:31 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi052=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: Saving service alertmanager spec with placement smithi052=a;count:1 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: pgmap v107: 97 pgs: 29 creating+peering, 56 active+clean, 12 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:50:32.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:31 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:50:33.325 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled grafana update... 2024-03-11T22:50:33.956 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: pgmap v110: 129 pgs: 29 creating+peering, 56 active+clean, 44 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T22:50:33.956 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:50:33.956 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:50:33.956 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:50:33.956 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:50:33.957 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:50:33.957 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:33 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:33.986 DEBUG:teuthology.orchestra.run.smithi116:grafana.a> sudo journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T22:50:33.988 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T22:50:33.989 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:50:34.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: pgmap v110: 129 pgs: 29 creating+peering, 56 active+clean, 44 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T22:50:34.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:50:34.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:50:34.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:50:34.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:50:34.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:50:34.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:33 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:34.283 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: pgmap v110: 129 pgs: 29 creating+peering, 56 active+clean, 44 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:50:34.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:33 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:35.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: Saving service grafana spec with placement smithi116=a;count:1 2024-03-11T22:50:35.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:50:35.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:50:35.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:50:35.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:34 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: Saving service grafana spec with placement smithi116=a;count:1 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:34 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi116=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:50:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: Saving service grafana spec with placement smithi116=a;count:1 2024-03-11T22:50:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2894738235' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:50:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/848000391' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:50:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:50:35.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:34 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: Deploying daemon prometheus.a on smithi116 2024-03-11T22:50:35.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: pgmap v113: 129 pgs: 20 creating+peering, 86 active+clean, 23 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:35.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:50:35.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: Cluster is now healthy 2024-03-11T22:50:35.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:35.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:35 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.156 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-03-11T22:50:36.156 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC8iu9lgXblCBAApLdFPy9d9TZ/ATolBZDrrQ== 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: Deploying daemon prometheus.a on smithi116 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: pgmap v113: 129 pgs: 20 creating+peering, 86 active+clean, 23 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: Cluster is now healthy 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:35 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: Deploying daemon prometheus.a on smithi116 2024-03-11T22:50:36.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: pgmap v113: 129 pgs: 20 creating+peering, 86 active+clean, 23 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:36.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:50:36.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: Cluster is now healthy 2024-03-11T22:50:36.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.330 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:35 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:36.829 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:50:36.829 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T22:50:36.829 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T22:50:36.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:50:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:36 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2391685037' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:37.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:36 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2391685037' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:37.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:36 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2391685037' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:37.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:36 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2391685037' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:37.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:36 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2391685037' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:37.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:36 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2391685037' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:37.553 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:50:38.187 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:37 smithi152 ceph-mon[27791]: pgmap v114: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-03-11T22:50:38.187 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:37 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.187 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:37 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.188 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:37 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:37 smithi116 ceph-mon[27552]: pgmap v114: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-03-11T22:50:38.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:37 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:37 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:37 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:37 smithi052 ceph-mon[25824]: pgmap v114: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.2 KiB/s wr, 140 op/s 2024-03-11T22:50:38.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:37 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:37 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:38.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:37 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:40.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:39 smithi116 ceph-mon[27552]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 5.3 KiB/s wr, 166 op/s 2024-03-11T22:50:40.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:39 smithi052 ceph-mon[25824]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 5.3 KiB/s wr, 166 op/s 2024-03-11T22:50:40.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:39 smithi152 ceph-mon[27791]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 5.3 KiB/s wr, 166 op/s 2024-03-11T22:50:41.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:41 smithi116 ceph-mon[27552]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:50:42.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:41 smithi052 ceph-mon[25824]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:50:42.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:41 smithi152 ceph-mon[27791]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:50:43.476 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2024-03-11T22:50:43.476 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQDDiu9lCizQGxAAqrqC7t7DSg5RCtbXGcYkSA== 2024-03-11T22:50:44.013 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:43 smithi116 ceph-mon[27552]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 115 op/s 2024-03-11T22:50:44.013 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:43 smithi116 ceph-mon[27552]: from='client.? 172.21.15.116:0/3500735430' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:44.013 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:43 smithi116 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:44.013 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:43 smithi116 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:44.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:43 smithi052 ceph-mon[25824]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 115 op/s 2024-03-11T22:50:44.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:43 smithi052 ceph-mon[25824]: from='client.? 172.21.15.116:0/3500735430' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:44.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:43 smithi052 ceph-mon[25824]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:44.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:43 smithi052 ceph-mon[25824]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:44.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:43 smithi152 ceph-mon[27791]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 115 op/s 2024-03-11T22:50:44.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:43 smithi152 ceph-mon[27791]: from='client.? 172.21.15.116:0/3500735430' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:44.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:43 smithi152 ceph-mon[27791]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:44.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:43 smithi152 ceph-mon[27791]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:46.235 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:45 smithi116 ceph-mon[27552]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 3.3 KiB/s wr, 104 op/s 2024-03-11T22:50:46.267 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:50:46.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T22:50:46.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T22:50:46.315 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:50:46.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:45 smithi052 ceph-mon[25824]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 3.3 KiB/s wr, 104 op/s 2024-03-11T22:50:46.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:45 smithi152 ceph-mon[27791]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 3.3 KiB/s wr, 104 op/s 2024-03-11T22:50:46.565 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:50:47.596 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 podman[35909]: 2024-03-11 22:50:47.48498486 +0000 UTC m=+0.122317614 container create be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 podman[35909]: 2024-03-11 22:50:47.686942962 +0000 UTC m=+0.324275727 container init be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 podman[35909]: 2024-03-11 22:50:47.692097552 +0000 UTC m=+0.329430306 container start be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.714Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.714Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.714Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi116 (none))" 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.715Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.715Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.717Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.717Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T22:50:47.947 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.718Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.718Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.719Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.719Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.588µs 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.719Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.719Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.719Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=14.49µs wal_replay_duration=150.023µs wbl_replay_duration=107ns total_replay_duration=181.737µs 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.720Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.720Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.720Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.737Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.257052ms db_storage=671ns remote_storage=1.249µs web_handler=260ns query_engine=665ns scrape=424.196µs scrape_sd=55.594µs notify=647ns notify_sd=1.224µs rules=16.604986ms tracing=5.617µs 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.737Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:50:47.737Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 bash[35909]: be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 2024-03-11T22:50:47.948 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:50:47 smithi116 systemd[1]: Started Ceph prometheus.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:50:47.995 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:47 smithi152 ceph-mon[27791]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:50:47.995 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:47 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:47.995 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:47 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:47.995 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:47 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:47.995 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:47 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:50:48.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:47 smithi052 ceph-mon[25824]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:50:48.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:47 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:48.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:47 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:48.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:47 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:48.235 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:47 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:50:48.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-mon[27552]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:50:48.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:48.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:48.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:48.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:47 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:50:48.497 INFO:teuthology.orchestra.run.smithi152.stdout:[client.2] 2024-03-11T22:50:48.497 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDIiu9lrzYhHRAAcFXoPqw/b0QQJHrAmsQKmg== 2024-03-11T22:50:49.079 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:48 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: ignoring --setuser ceph since I am not root 2024-03-11T22:50:49.079 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:48 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: ignoring --setgroup ceph since I am not root 2024-03-11T22:50:49.079 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:48 smithi152 ceph-mgr[30297]: -- 172.21.15.152:0/1831167058 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f6825d61a0 con 0x55f6825ad000 2024-03-11T22:50:49.167 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:50:49.167 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-11T22:50:49.167 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-11T22:50:49.225 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.225 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.225 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.225 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.225 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='client.? 172.21.15.152:0/3847768666' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-mon[27552]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:48 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: ignoring --setuser ceph since I am not root 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:48 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: ignoring --setgroup ceph since I am not root 2024-03-11T22:50:49.226 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:48 smithi116 ceph-mgr[29272]: -- 172.21.15.116:0/3784870081 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5621ab9d61a0 con 0x5621ab9ac800 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='client.? 172.21.15.152:0/3847768666' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:49.230 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:50:49.231 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-mon[25824]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:50:49.231 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:48 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ignoring --setuser ceph since I am not root 2024-03-11T22:50:49.231 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:48 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ignoring --setgroup ceph since I am not root 2024-03-11T22:50:49.252 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T22:50:49.252 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T22:50:49.253 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='client.? 172.21.15.152:0/3847768666' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:50:49.413 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:50:49.414 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: from='mgr.14152 172.21.15.52:0/445790985' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:50:49.414 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-mon[27791]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:50:49.487 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:49.449+0000 7f236ab2b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:50:49.504 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:49.761 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:49.478+0000 7f0e0fc31200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:50:49.761 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:49 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:49.582+0000 7f0e0fc31200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:50:49.829 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:49.471+0000 7f07db678200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:50:49.829 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:49 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:49.573+0000 7f07db678200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:50:49.987 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:49 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:49.544+0000 7f236ab2b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:50:50.320 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:50 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:50.252+0000 7f0e0fc31200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:50:50.487 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.171+0000 7f236ab2b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:50:50.487 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.264+0000 7f236ab2b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:50:50.487 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.403+0000 7f236ab2b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:50:50.575 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:50 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:50.235+0000 7f07db678200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:50:50.575 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:50 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:50.330+0000 7f07db678200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:50:50.575 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:50 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:50.473+0000 7f07db678200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:50:50.575 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:50 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:50.568+0000 7f07db678200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:50:50.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:50 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:50.347+0000 7f0e0fc31200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:50:50.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:50 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:50.496+0000 7f0e0fc31200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:50:50.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:50 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:50.595+0000 7f0e0fc31200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:50:50.687 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:50 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:50.680+0000 7f0e0fc31200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:50:50.829 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:50 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:50.652+0000 7f07db678200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:50:50.845 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.497+0000 7f236ab2b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:50:50.845 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.579+0000 7f236ab2b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:50:51.132 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.839+0000 7f236ab2b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:50:51.132 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:50 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:50.949+0000 7f236ab2b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:50:51.132 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:51.034+0000 7f236ab2b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:50:51.208 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:50 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:50.915+0000 7f07db678200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:50:51.208 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:51.023+0000 7f07db678200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:50:51.208 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:51.108+0000 7f07db678200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:50:51.208 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:51.202+0000 7f07db678200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:50:51.286 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:50 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:50.971+0000 7f0e0fc31200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:50:51.286 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:51.090+0000 7f0e0fc31200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:50:51.286 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:51.179+0000 7f0e0fc31200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:50:51.386 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:51.126+0000 7f236ab2b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:50:51.386 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:51.310+0000 7f236ab2b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:50:51.566 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:50:51.567 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:51.280+0000 7f0e0fc31200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:50:51.567 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:51.464+0000 7f0e0fc31200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:50:51.579 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:51.380+0000 7f07db678200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:50:51.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:51 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2527518866' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:50:51.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:51 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2527518866' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:50:52.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:51 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2527518866' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:50:52.228 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":3606840060},{"type":"v1","addr":"172.21.15.52:6801","nonce":3606840060}]},"active_addr":"172.21.15.52:6801/3606840060","active_change":"2024-03-11T22:47:38.034042+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.52:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2376232073}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2354249853}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2373998825}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1678161314}]}]} 2024-03-11T22:50:52.234 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T22:50:52.234 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:50:52.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:50:52.484 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:52.737 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:52 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:52.298+0000 7f236ab2b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:50:52.737 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:52 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:52.429+0000 7f236ab2b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:50:52.820 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:52.335+0000 7f07db678200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:50:52.820 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:52.462+0000 7f07db678200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:50:52.820 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:52.814+0000 7f07db678200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:50:52.820 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:52.493+0000 7f0e0fc31200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:50:52.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:52.631+0000 7f0e0fc31200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:50:53.044 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:52 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:52.755+0000 7f236ab2b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:50:53.045 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:52 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:52.844+0000 7f236ab2b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:50:53.045 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:53.037+0000 7f236ab2b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:50:53.079 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:52 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:52.903+0000 7f07db678200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:50:53.283 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:52 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:52.979+0000 7f0e0fc31200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:50:53.283 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:53.073+0000 7f0e0fc31200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:50:53.283 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:53.277+0000 7f0e0fc31200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:50:53.487 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:53.220+0000 7f236ab2b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:50:53.570 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:53.469+0000 7f0e0fc31200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:50:53.571 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:53 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:53.098+0000 7f07db678200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:50:53.571 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:53 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:53.282+0000 7f07db678200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:50:53.763 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:53.502+0000 7f236ab2b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:50:53.829 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:53 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:53.565+0000 7f07db678200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:50:53.895 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:53 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:53.769+0000 7f0e0fc31200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:50:54.237 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:53.756+0000 7f236ab2b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:50:54.237 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:53 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:53.987+0000 7f236ab2b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:50:54.294 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:54 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:54.037+0000 7f0e0fc31200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:50:54.294 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:54 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:54.287+0000 7f0e0fc31200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:50:54.329 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:53 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:53.825+0000 7f07db678200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:50:54.329 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:54.060+0000 7f07db678200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:50:54.348 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:50:54.348 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":31,"fsid":"09234c40-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:46:44.273030+0000","modified":"2024-03-11T22:50:33.907436+0000","last_up_change":"2024-03-11T22:50:12.635056+0000","last_in_change":"2024-03-11T22:49:51.280321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:50:14.378883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T22:50:25.202126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:50:27.211811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:50:29.158247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:50:31.224794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63880b10-467d-407b-8c32-d986c0a2e54a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6803","nonce":161116271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6805","nonce":161116271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6809","nonce":161116271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6807","nonce":161116271}]},"public_addr":"172.21.15.52:6803/161116271","cluster_addr":"172.21.15.52:6805/161116271","heartbeat_back_addr":"172.21.15.52:6809/161116271","heartbeat_front_addr":"172.21.15.52:6807/161116271","state":["exists","up"]},{"osd":1,"uuid":"74ef0290-69e4-4bf0-89d6-92e9a43d5055","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6801","nonce":1231550709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6803","nonce":1231550709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6807","nonce":1231550709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6805","nonce":1231550709}]},"public_addr":"172.21.15.116:6801/1231550709","cluster_addr":"172.21.15.116:6803/1231550709","heartbeat_back_addr":"172.21.15.116:6807/1231550709","heartbeat_front_addr":"172.21.15.116:6805/1231550709","state":["exists","up"]},{"osd":2,"uuid":"0c88b7ec-27d6-4f07-b2ae-aca49f3afa35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6801","nonce":3140177155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6803","nonce":3140177155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6807","nonce":3140177155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6805","nonce":3140177155}]},"public_addr":"172.21.15.152:6801/3140177155","cluster_addr":"172.21.15.152:6803/3140177155","heartbeat_back_addr":"172.21.15.152:6807/3140177155","heartbeat_front_addr":"172.21.15.152:6805/3140177155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:49:17.187764+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:49:43.057891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:50:10.637190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:0/2870261962":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6801/1990955331":"2024-03-12T22:47:38.033876+0000","172.21.15.52:0/3377302881":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6800/1990955331":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6801/3464055426":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/664002337":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6800/3464055426":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/1304186236":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/1596447173":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/3021375888":"2024-03-12T22:47:11.284060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:50:54.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:54 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2309305632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:54.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:54 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2309305632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:54.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:54 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2309305632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:54.975 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:50:54.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:50:55.218 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:55.237 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:54 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:54.768+0000 7f236ab2b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:50:55.320 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:55.136+0000 7f0e0fc31200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:50:55.329 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:54 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:54.853+0000 7f07db678200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:50:55.652 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:55.331+0000 7f236ab2b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:50:55.652 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:55.429+0000 7f236ab2b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:50:55.652 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:55.645+0000 7f236ab2b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:50:55.726 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:55.429+0000 7f07db678200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:50:55.726 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:55.545+0000 7f07db678200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-mon[27552]: Standby manager daemon x restarted 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-mon[27552]: Standby manager daemon x started 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: [11/Mar/2024:22:50:55] ENGINE Bus STARTING 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: CherryPy Checker: 2024-03-11T22:50:55.964 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: The Application mounted at '' has an empty config. 2024-03-11T22:50:55.965 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: 2024-03-11T22:50:55.965 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: [11/Mar/2024:22:50:55] ENGINE Serving on http://:::9283 2024-03-11T22:50:55.965 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:50:55 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x[29248]: [11/Mar/2024:22:50:55] ENGINE Bus STARTED 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:55.742+0000 7f0e0fc31200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:55.845+0000 7f0e0fc31200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-mon[25824]: Standby manager daemon x restarted 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-mon[25824]: Standby manager daemon x started 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:50:56.038 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:50:56.039 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:55 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:50:56.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-mon[27791]: Standby manager daemon x restarted 2024-03-11T22:50:56.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-mon[27791]: Standby manager daemon x started 2024-03-11T22:50:56.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:50:56.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.116:0/1352103297' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:55.783+0000 7f07db678200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: [11/Mar/2024:22:50:55] ENGINE Bus STARTING 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: CherryPy Checker: 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: The Application mounted at '' has an empty config. 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: [11/Mar/2024:22:50:55] ENGINE Serving on http://:::9283 2024-03-11T22:50:56.080 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:50:55 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z[30240]: [11/Mar/2024:22:50:55] ENGINE Bus STARTED 2024-03-11T22:50:56.317 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:56.073+0000 7f0e0fc31200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:50:56.318 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:50:56] ENGINE Bus STARTING 2024-03-11T22:50:56.570 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: CherryPy Checker: 2024-03-11T22:50:56.571 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: The Application mounted at '' has an empty config. 2024-03-11T22:50:56.571 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:50:56.571 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:50:56] ENGINE Serving on http://:::9283 2024-03-11T22:50:56.571 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:50:56] ENGINE Bus STARTED 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: Standby manager daemon z restarted 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: Standby manager daemon z started 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: Active manager daemon y restarted 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: Activating manager daemon y 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T22:50:56.789 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: mgrmap e19: y(active, starting, since 0.00961089s), standbys: z, x 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:50:56.790 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: Manager daemon y is now available 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:50:56.791 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:50:56.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: Standby manager daemon z restarted 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: Standby manager daemon z started 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: Active manager daemon y restarted 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: Activating manager daemon y 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: mgrmap e19: y(active, starting, since 0.00961089s), standbys: z, x 2024-03-11T22:50:56.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: Manager daemon y is now available 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:56.831 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:50:56.832 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:50:56.952 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: Standby manager daemon z restarted 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: Standby manager daemon z started 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.? 172.21.15.152:0/1258865945' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: Active manager daemon y restarted 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: Activating manager daemon y 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: mgrmap e19: y(active, starting, since 0.00961089s), standbys: z, x 2024-03-11T22:50:56.953 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: Manager daemon y is now available 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:56.954 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi052", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]': finished 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:50:56.955 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:50:57.198 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:50:57.199 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":32,"fsid":"09234c40-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:46:44.273030+0000","modified":"2024-03-11T22:50:56.088092+0000","last_up_change":"2024-03-11T22:50:12.635056+0000","last_in_change":"2024-03-11T22:49:51.280321+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:50:14.378883+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T22:50:25.202126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:50:27.211811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:50:29.158247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:50:31.224794+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63880b10-467d-407b-8c32-d986c0a2e54a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6803","nonce":161116271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6805","nonce":161116271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6809","nonce":161116271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":161116271},{"type":"v1","addr":"172.21.15.52:6807","nonce":161116271}]},"public_addr":"172.21.15.52:6803/161116271","cluster_addr":"172.21.15.52:6805/161116271","heartbeat_back_addr":"172.21.15.52:6809/161116271","heartbeat_front_addr":"172.21.15.52:6807/161116271","state":["exists","up"]},{"osd":1,"uuid":"74ef0290-69e4-4bf0-89d6-92e9a43d5055","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6801","nonce":1231550709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6803","nonce":1231550709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6807","nonce":1231550709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1231550709},{"type":"v1","addr":"172.21.15.116:6805","nonce":1231550709}]},"public_addr":"172.21.15.116:6801/1231550709","cluster_addr":"172.21.15.116:6803/1231550709","heartbeat_back_addr":"172.21.15.116:6807/1231550709","heartbeat_front_addr":"172.21.15.116:6805/1231550709","state":["exists","up"]},{"osd":2,"uuid":"0c88b7ec-27d6-4f07-b2ae-aca49f3afa35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6801","nonce":3140177155}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6803","nonce":3140177155}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6807","nonce":3140177155}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3140177155},{"type":"v1","addr":"172.21.15.152:6805","nonce":3140177155}]},"public_addr":"172.21.15.152:6801/3140177155","cluster_addr":"172.21.15.152:6803/3140177155","heartbeat_back_addr":"172.21.15.152:6807/3140177155","heartbeat_front_addr":"172.21.15.152:6805/3140177155","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:49:17.187764+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:49:43.057891+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:50:10.637190+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.52:6800/3606840060":"2024-03-12T22:50:56.088061+0000","172.21.15.52:0/2373998825":"2024-03-12T22:50:56.088061+0000","172.21.15.52:0/1678161314":"2024-03-12T22:50:56.088061+0000","172.21.15.52:0/2354249853":"2024-03-12T22:50:56.088061+0000","172.21.15.52:0/2870261962":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6801/1990955331":"2024-03-12T22:47:38.033876+0000","172.21.15.52:0/3377302881":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6800/1990955331":"2024-03-12T22:47:38.033876+0000","172.21.15.52:0/2376232073":"2024-03-12T22:50:56.088061+0000","172.21.15.52:6801/3464055426":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/664002337":"2024-03-12T22:47:38.033876+0000","172.21.15.52:6801/3606840060":"2024-03-12T22:50:56.088061+0000","172.21.15.52:6800/3464055426":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/1304186236":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/1596447173":"2024-03-12T22:47:11.284060+0000","172.21.15.52:0/3021375888":"2024-03-12T22:47:11.284060+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: mgrmap e20: y(active, since 1.02106s), standbys: z, x 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/4190817312' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:57.917 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.918 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:57 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:50:57.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:50:57.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: mgrmap e20: y(active, since 1.02106s), standbys: z, x 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/4190817312' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:57.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:57.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:57 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: Updating smithi052:/etc/ceph/ceph.conf 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: mgrmap e20: y(active, since 1.02106s), standbys: z, x 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/4190817312' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.080 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.081 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:57 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:50:58.219 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:58.221 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:58.221 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:50:58.898 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: [11/Mar/2024:22:50:57] ENGINE Bus STARTING 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: [11/Mar/2024:22:50:57] ENGINE Serving on http://172.21.15.52:8765 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: [11/Mar/2024:22:50:57] ENGINE Serving on https://172.21.15.52:7151 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: [11/Mar/2024:22:50:57] ENGINE Bus STARTED 2024-03-11T22:50:58.899 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:58 smithi152 ceph-mon[27791]: Deploying daemon alertmanager.a on smithi052 2024-03-11T22:50:58.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: [11/Mar/2024:22:50:57] ENGINE Bus STARTING 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: [11/Mar/2024:22:50:57] ENGINE Serving on http://172.21.15.52:8765 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: [11/Mar/2024:22:50:57] ENGINE Serving on https://172.21.15.52:7151 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: [11/Mar/2024:22:50:57] ENGINE Bus STARTED 2024-03-11T22:50:58.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:58 smithi116 ceph-mon[27552]: Deploying daemon alertmanager.a on smithi052 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Updating smithi052:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: [11/Mar/2024:22:50:57] ENGINE Bus STARTING 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: [11/Mar/2024:22:50:57] ENGINE Serving on http://172.21.15.52:8765 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Updating smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Updating smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Updating smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:50:58.990 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: [11/Mar/2024:22:50:57] ENGINE Serving on https://172.21.15.52:7151 2024-03-11T22:50:58.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: [11/Mar/2024:22:50:57] ENGINE Bus STARTED 2024-03-11T22:50:58.991 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:58 smithi052 ceph-mon[25824]: Deploying daemon alertmanager.a on smithi052 2024-03-11T22:50:59.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:59 smithi116 ceph-mon[27552]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:50:59.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:50:59 smithi116 ceph-mon[27552]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-11T22:51:00.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:59 smithi052 ceph-mon[25824]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:50:59 smithi052 ceph-mon[25824]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-11T22:51:00.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:59 smithi152 ceph-mon[27791]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:00.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:50:59 smithi152 ceph-mon[27791]: mgrmap e21: y(active, since 3s), standbys: z, x 2024-03-11T22:51:01.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:01 smithi052 ceph-mon[25824]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:01.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:01 smithi052 ceph-mon[25824]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-11T22:51:01.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:01 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:51:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:51:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:01 smithi116 ceph-mon[27552]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:01 smithi116 ceph-mon[27552]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-11T22:51:02.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:01 smithi152 ceph-mon[27791]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:02.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:01 smithi152 ceph-mon[27791]: mgrmap e22: y(active, since 4s), standbys: z, x 2024-03-11T22:51:04.028 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:03 smithi052 ceph-mon[25824]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:04.054 INFO:teuthology.orchestra.run.smithi052.stdout:34359738391 2024-03-11T22:51:04.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:51:04.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:03 smithi152 ceph-mon[27791]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:04.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:03 smithi116 ceph-mon[27552]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:05.332 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:05.783 INFO:teuthology.orchestra.run.smithi052.stdout:77309411341 2024-03-11T22:51:05.784 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:51:06.040 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:05 smithi052 ceph-mon[25824]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:51:06.077 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:05 smithi116 ceph-mon[27552]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:51:06.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:05 smithi152 ceph-mon[27791]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:51:07.980 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:07 smithi052 ceph-mon[25824]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:51:08.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:07 smithi152 ceph-mon[27791]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:51:08.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:07 smithi116 ceph-mon[27552]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:51:09.348 INFO:teuthology.orchestra.run.smithi052.stdout:55834574867 2024-03-11T22:51:09.348 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:51:10.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:09 smithi052 ceph-mon[25824]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:51:10.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:09 smithi152 ceph-mon[27791]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:51:10.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:09 smithi116 ceph-mon[27552]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:51:10.957 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:11.750 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:11 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:51:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:51:12.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:11 smithi052 ceph-mon[25824]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:12.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:11 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:12.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:11 smithi152 ceph-mon[27791]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:12.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:11 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:12.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:11 smithi116 ceph-mon[27552]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:12.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:11 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:12.504 INFO:teuthology.orchestra.run.smithi052.stdout:34359738392 2024-03-11T22:51:12.754 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:12 smithi052 systemd[1]: Starting Ceph alertmanager.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:51:13.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:12 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2986930703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:51:13.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:12 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2986930703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:51:13.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:12 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2986930703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:51:13.930 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:13 smithi052 ceph-mon[25824]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:14.027 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:14.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:13 smithi152 ceph-mon[27791]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:14.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:13 smithi116 ceph-mon[27552]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:14.266 INFO:teuthology.orchestra.run.smithi052.stdout:77309411342 2024-03-11T22:51:14.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:14 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/2418414998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:51:15.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:14 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/2418414998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:51:15.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:14 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/2418414998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:51:15.456 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738391 got 34359738392 for osd.0 2024-03-11T22:51:15.456 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:16.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:15 smithi052 ceph-mon[25824]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:16.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:15 smithi152 ceph-mon[27791]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:16.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411341 got 77309411342 for osd.2 2024-03-11T22:51:16.174 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:16.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:15 smithi116 ceph-mon[27552]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:51:16.836 INFO:teuthology.orchestra.run.smithi052.stdout:55834574868 2024-03-11T22:51:17.067 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:16 smithi052 podman[48057]: 2024-03-11 22:51:16.609882414 +0000 UTC m=+1.150011149 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T22:51:17.067 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:16 smithi052 podman[48057]: 2024-03-11 22:51:16.716091788 +0000 UTC m=+1.256220510 volume create 026ecc2079ee9ad888caffa275da05bd75d2678b3363f19988786d3cdbd62620 2024-03-11T22:51:17.067 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:16 smithi052 podman[48057]: 2024-03-11 22:51:16.816519445 +0000 UTC m=+1.356648168 container create 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:51:17.563 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574867 got 55834574868 for osd.1 2024-03-11T22:51:17.563 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:17.564 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:51:17.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:51:17.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-mon[25824]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:17.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/174416333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 podman[48057]: 2024-03-11 22:51:17.714059826 +0000 UTC m=+2.254188549 container init 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 podman[48057]: 2024-03-11 22:51:17.719844014 +0000 UTC m=+2.259972743 container start 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.733Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.734Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.735Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.52 port=9094 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.736Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.760Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.761Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.763Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T22:51:17.821 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:17.764Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T22:51:17.895 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:17 smithi116 ceph-mon[27552]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:17 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/174416333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:51:18.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:17 smithi152 ceph-mon[27791]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:18.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:17 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/174416333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:51:18.133 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 bash[48057]: 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec 2024-03-11T22:51:18.133 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:17 smithi052 systemd[1]: Started Ceph alertmanager.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:51:18.133 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:51:18] ENGINE Bus STOPPING 2024-03-11T22:51:18.134 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:51:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:51:18.134 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:51:18] ENGINE Bus STOPPED 2024-03-11T22:51:18.134 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:51:18] ENGINE Bus STARTING 2024-03-11T22:51:18.402 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:51:18] ENGINE Serving on http://:::9283 2024-03-11T22:51:18.402 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:51:18] ENGINE Bus STARTED 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:51:19.041 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:18 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.143 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:51:19.144 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:18 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:51:19.238 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:18 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:19.885 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:19 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:19.736Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000187473s 2024-03-11T22:51:19.965 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:19 smithi152 ceph-mon[27791]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:51:19.965 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:19 smithi152 ceph-mon[27791]: Deploying daemon grafana.a on smithi116 2024-03-11T22:51:19.965 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:19 smithi152 ceph-mon[27791]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:20.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:19 smithi116 ceph-mon[27552]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:51:20.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:19 smithi116 ceph-mon[27552]: Deploying daemon grafana.a on smithi116 2024-03-11T22:51:20.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:19 smithi116 ceph-mon[27552]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:20.266 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:19 smithi052 ceph-mon[25824]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:51:20.266 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:19 smithi052 ceph-mon[25824]: Deploying daemon grafana.a on smithi116 2024-03-11T22:51:20.266 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:19 smithi052 ceph-mon[25824]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:21.607 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:21 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:51:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:51:21.910 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:21 smithi052 ceph-mon[25824]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:21.911 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:21 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:22.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:21 smithi116 ceph-mon[27552]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:22.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:21 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:22.243 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:51:22.246 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-03-11T22:51:22.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:21 smithi152 ceph-mon[27791]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:22.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:21 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:23.111 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-03-11T22:51:22.108179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":100808,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":281122360,"statfs":{"total":287972524032,"available":287869296640,"internally_reserved":0,"allocated":3182592,"data_stored":2109498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002676"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095416+0000","last_change":"2024-03-11T22:50:32.923509+0000","last_active":"2024-03-11T22:50:56.095416+0000","last_peered":"2024-03-11T22:50:56.095416+0000","last_clean":"2024-03-11T22:50:56.095416+0000","last_became_active":"2024-03-11T22:50:32.923169+0000","last_became_peered":"2024-03-11T22:50:32.923169+0000","last_unstale":"2024-03-11T22:50:56.095416+0000","last_undegraded":"2024-03-11T22:50:56.095416+0000","last_fullsized":"2024-03-11T22:50:56.095416+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:07.362782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094190+0000","last_change":"2024-03-11T22:50:30.921468+0000","last_active":"2024-03-11T22:50:56.094190+0000","last_peered":"2024-03-11T22:50:56.094190+0000","last_clean":"2024-03-11T22:50:56.094190+0000","last_became_active":"2024-03-11T22:50:30.919382+0000","last_became_peered":"2024-03-11T22:50:30.919382+0000","last_unstale":"2024-03-11T22:50:56.094190+0000","last_undegraded":"2024-03-11T22:50:56.094190+0000","last_fullsized":"2024-03-11T22:50:56.094190+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:22.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095504+0000","last_change":"2024-03-11T22:50:26.899261+0000","last_active":"2024-03-11T22:50:56.095504+0000","last_peered":"2024-03-11T22:50:56.095504+0000","last_clean":"2024-03-11T22:50:56.095504+0000","last_became_active":"2024-03-11T22:50:26.898518+0000","last_became_peered":"2024-03-11T22:50:26.898518+0000","last_unstale":"2024-03-11T22:50:56.095504+0000","last_undegraded":"2024-03-11T22:50:56.095504+0000","last_fullsized":"2024-03-11T22:50:56.095504+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:34:25.851856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093659+0000","last_change":"2024-03-11T22:50:28.909151+0000","last_active":"2024-03-11T22:50:56.093659+0000","last_peered":"2024-03-11T22:50:56.093659+0000","last_clean":"2024-03-11T22:50:56.093659+0000","last_became_active":"2024-03-11T22:50:28.907933+0000","last_became_peered":"2024-03-11T22:50:28.907933+0000","last_unstale":"2024-03-11T22:50:56.093659+0000","last_undegraded":"2024-03-11T22:50:56.093659+0000","last_fullsized":"2024-03-11T22:50:56.093659+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:17:05.778171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093742+0000","last_change":"2024-03-11T22:50:32.919666+0000","last_active":"2024-03-11T22:50:56.093742+0000","last_peered":"2024-03-11T22:50:56.093742+0000","last_clean":"2024-03-11T22:50:56.093742+0000","last_became_active":"2024-03-11T22:50:32.919336+0000","last_became_peered":"2024-03-11T22:50:32.919336+0000","last_unstale":"2024-03-11T22:50:56.093742+0000","last_undegraded":"2024-03-11T22:50:56.093742+0000","last_fullsized":"2024-03-11T22:50:56.093742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:35:55.927559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095040+0000","last_change":"2024-03-11T22:50:30.920136+0000","last_active":"2024-03-11T22:50:56.095040+0000","last_peered":"2024-03-11T22:50:56.095040+0000","last_clean":"2024-03-11T22:50:56.095040+0000","last_became_active":"2024-03-11T22:50:30.919819+0000","last_became_peered":"2024-03-11T22:50:30.919819+0000","last_unstale":"2024-03-11T22:50:56.095040+0000","last_undegraded":"2024-03-11T22:50:56.095040+0000","last_fullsized":"2024-03-11T22:50:56.095040+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:50.741318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094281+0000","last_change":"2024-03-11T22:50:26.900519+0000","last_active":"2024-03-11T22:50:56.094281+0000","last_peered":"2024-03-11T22:50:56.094281+0000","last_clean":"2024-03-11T22:50:56.094281+0000","last_became_active":"2024-03-11T22:50:26.900179+0000","last_became_peered":"2024-03-11T22:50:26.900179+0000","last_unstale":"2024-03-11T22:50:56.094281+0000","last_undegraded":"2024-03-11T22:50:56.094281+0000","last_fullsized":"2024-03-11T22:50:56.094281+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:29:32.932049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094726+0000","last_change":"2024-03-11T22:50:28.910400+0000","last_active":"2024-03-11T22:50:56.094726+0000","last_peered":"2024-03-11T22:50:56.094726+0000","last_clean":"2024-03-11T22:50:56.094726+0000","last_became_active":"2024-03-11T22:50:28.909646+0000","last_became_peered":"2024-03-11T22:50:28.909646+0000","last_unstale":"2024-03-11T22:50:56.094726+0000","last_undegraded":"2024-03-11T22:50:56.094726+0000","last_fullsized":"2024-03-11T22:50:56.094726+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:26:28.676129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094309+0000","last_change":"2024-03-11T22:50:32.916141+0000","last_active":"2024-03-11T22:50:56.094309+0000","last_peered":"2024-03-11T22:50:56.094309+0000","last_clean":"2024-03-11T22:50:56.094309+0000","last_became_active":"2024-03-11T22:50:32.915812+0000","last_became_peered":"2024-03-11T22:50:32.915812+0000","last_unstale":"2024-03-11T22:50:56.094309+0000","last_undegraded":"2024-03-11T22:50:56.094309+0000","last_fullsized":"2024-03-11T22:50:56.094309+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:30.646032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096057+0000","last_change":"2024-03-11T22:50:30.919789+0000","last_active":"2024-03-11T22:50:56.096057+0000","last_peered":"2024-03-11T22:50:56.096057+0000","last_clean":"2024-03-11T22:50:56.096057+0000","last_became_active":"2024-03-11T22:50:30.919444+0000","last_became_peered":"2024-03-11T22:50:30.919444+0000","last_unstale":"2024-03-11T22:50:56.096057+0000","last_undegraded":"2024-03-11T22:50:56.096057+0000","last_fullsized":"2024-03-11T22:50:56.096057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:58.773034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097370+0000","last_change":"2024-03-11T22:50:26.903897+0000","last_active":"2024-03-11T22:50:56.097370+0000","last_peered":"2024-03-11T22:50:56.097370+0000","last_clean":"2024-03-11T22:50:56.097370+0000","last_became_active":"2024-03-11T22:50:26.902960+0000","last_became_peered":"2024-03-11T22:50:26.902960+0000","last_unstale":"2024-03-11T22:50:56.097370+0000","last_undegraded":"2024-03-11T22:50:56.097370+0000","last_fullsized":"2024-03-11T22:50:56.097370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:30.358238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095365+0000","last_change":"2024-03-11T22:50:28.910110+0000","last_active":"2024-03-11T22:50:56.095365+0000","last_peered":"2024-03-11T22:50:56.095365+0000","last_clean":"2024-03-11T22:50:56.095365+0000","last_became_active":"2024-03-11T22:50:28.908945+0000","last_became_peered":"2024-03-11T22:50:28.908945+0000","last_unstale":"2024-03-11T22:50:56.095365+0000","last_undegraded":"2024-03-11T22:50:56.095365+0000","last_fullsized":"2024-03-11T22:50:56.095365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:51:05.096198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093805+0000","last_change":"2024-03-11T22:50:32.919705+0000","last_active":"2024-03-11T22:50:56.093805+0000","last_peered":"2024-03-11T22:50:56.093805+0000","last_clean":"2024-03-11T22:50:56.093805+0000","last_became_active":"2024-03-11T22:50:32.919260+0000","last_became_peered":"2024-03-11T22:50:32.919260+0000","last_unstale":"2024-03-11T22:50:56.093805+0000","last_undegraded":"2024-03-11T22:50:56.093805+0000","last_fullsized":"2024-03-11T22:50:56.093805+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:23:00.848480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093742+0000","last_change":"2024-03-11T22:50:30.916838+0000","last_active":"2024-03-11T22:50:56.093742+0000","last_peered":"2024-03-11T22:50:56.093742+0000","last_clean":"2024-03-11T22:50:56.093742+0000","last_became_active":"2024-03-11T22:50:30.915877+0000","last_became_peered":"2024-03-11T22:50:30.915877+0000","last_unstale":"2024-03-11T22:50:56.093742+0000","last_undegraded":"2024-03-11T22:50:56.093742+0000","last_fullsized":"2024-03-11T22:50:56.093742+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:40:20.022725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095658+0000","last_change":"2024-03-11T22:50:26.901853+0000","last_active":"2024-03-11T22:50:56.095658+0000","last_peered":"2024-03-11T22:50:56.095658+0000","last_clean":"2024-03-11T22:50:56.095658+0000","last_became_active":"2024-03-11T22:50:26.901490+0000","last_became_peered":"2024-03-11T22:50:26.901490+0000","last_unstale":"2024-03-11T22:50:56.095658+0000","last_undegraded":"2024-03-11T22:50:56.095658+0000","last_fullsized":"2024-03-11T22:50:56.095658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:41:05.428025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097241+0000","last_change":"2024-03-11T22:50:28.913248+0000","last_active":"2024-03-11T22:50:56.097241+0000","last_peered":"2024-03-11T22:50:56.097241+0000","last_clean":"2024-03-11T22:50:56.097241+0000","last_became_active":"2024-03-11T22:50:28.912856+0000","last_became_peered":"2024-03-11T22:50:28.912856+0000","last_unstale":"2024-03-11T22:50:56.097241+0000","last_undegraded":"2024-03-11T22:50:56.097241+0000","last_fullsized":"2024-03-11T22:50:56.097241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:25:29.487389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094503+0000","last_change":"2024-03-11T22:50:32.920490+0000","last_active":"2024-03-11T22:50:56.094503+0000","last_peered":"2024-03-11T22:50:56.094503+0000","last_clean":"2024-03-11T22:50:56.094503+0000","last_became_active":"2024-03-11T22:50:32.920123+0000","last_became_peered":"2024-03-11T22:50:32.920123+0000","last_unstale":"2024-03-11T22:50:56.094503+0000","last_undegraded":"2024-03-11T22:50:56.094503+0000","last_fullsized":"2024-03-11T22:50:56.094503+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:57.485435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095981+0000","last_change":"2024-03-11T22:50:30.921770+0000","last_active":"2024-03-11T22:50:56.095981+0000","last_peered":"2024-03-11T22:50:56.095981+0000","last_clean":"2024-03-11T22:50:56.095981+0000","last_became_active":"2024-03-11T22:50:30.921441+0000","last_became_peered":"2024-03-11T22:50:30.921441+0000","last_unstale":"2024-03-11T22:50:56.095981+0000","last_undegraded":"2024-03-11T22:50:56.095981+0000","last_fullsized":"2024-03-11T22:50:56.095981+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:21:45.963299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095412+0000","last_change":"2024-03-11T22:50:26.899801+0000","last_active":"2024-03-11T22:50:56.095412+0000","last_peered":"2024-03-11T22:50:56.095412+0000","last_clean":"2024-03-11T22:50:56.095412+0000","last_became_active":"2024-03-11T22:50:26.899445+0000","last_became_peered":"2024-03-11T22:50:26.899445+0000","last_unstale":"2024-03-11T22:50:56.095412+0000","last_undegraded":"2024-03-11T22:50:56.095412+0000","last_fullsized":"2024-03-11T22:50:56.095412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:01:58.268382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095484+0000","last_change":"2024-03-11T22:50:28.912720+0000","last_active":"2024-03-11T22:50:56.095484+0000","last_peered":"2024-03-11T22:50:56.095484+0000","last_clean":"2024-03-11T22:50:56.095484+0000","last_became_active":"2024-03-11T22:50:28.912422+0000","last_became_peered":"2024-03-11T22:50:28.912422+0000","last_unstale":"2024-03-11T22:50:56.095484+0000","last_undegraded":"2024-03-11T22:50:56.095484+0000","last_fullsized":"2024-03-11T22:50:56.095484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:36.835856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095617+0000","last_change":"2024-03-11T22:50:32.921678+0000","last_active":"2024-03-11T22:50:56.095617+0000","last_peered":"2024-03-11T22:50:56.095617+0000","last_clean":"2024-03-11T22:50:56.095617+0000","last_became_active":"2024-03-11T22:50:32.921260+0000","last_became_peered":"2024-03-11T22:50:32.921260+0000","last_unstale":"2024-03-11T22:50:56.095617+0000","last_undegraded":"2024-03-11T22:50:56.095617+0000","last_fullsized":"2024-03-11T22:50:56.095617+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:38:33.220835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095596+0000","last_change":"2024-03-11T22:50:30.920908+0000","last_active":"2024-03-11T22:50:56.095596+0000","last_peered":"2024-03-11T22:50:56.095596+0000","last_clean":"2024-03-11T22:50:56.095596+0000","last_became_active":"2024-03-11T22:50:30.920104+0000","last_became_peered":"2024-03-11T22:50:30.920104+0000","last_unstale":"2024-03-11T22:50:56.095596+0000","last_undegraded":"2024-03-11T22:50:56.095596+0000","last_fullsized":"2024-03-11T22:50:56.095596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:28.236707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094615+0000","last_change":"2024-03-11T22:50:26.902820+0000","last_active":"2024-03-11T22:50:56.094615+0000","last_peered":"2024-03-11T22:50:56.094615+0000","last_clean":"2024-03-11T22:50:56.094615+0000","last_became_active":"2024-03-11T22:50:26.902245+0000","last_became_peered":"2024-03-11T22:50:26.902245+0000","last_unstale":"2024-03-11T22:50:56.094615+0000","last_undegraded":"2024-03-11T22:50:56.094615+0000","last_fullsized":"2024-03-11T22:50:56.094615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:06.302124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095137+0000","last_change":"2024-03-11T22:50:28.909065+0000","last_active":"2024-03-11T22:50:56.095137+0000","last_peered":"2024-03-11T22:50:56.095137+0000","last_clean":"2024-03-11T22:50:56.095137+0000","last_became_active":"2024-03-11T22:50:28.908728+0000","last_became_peered":"2024-03-11T22:50:28.908728+0000","last_unstale":"2024-03-11T22:50:56.095137+0000","last_undegraded":"2024-03-11T22:50:56.095137+0000","last_fullsized":"2024-03-11T22:50:56.095137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:21.757165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093244+0000","last_change":"2024-03-11T22:50:32.922608+0000","last_active":"2024-03-11T22:50:56.093244+0000","last_peered":"2024-03-11T22:50:56.093244+0000","last_clean":"2024-03-11T22:50:56.093244+0000","last_became_active":"2024-03-11T22:50:32.921834+0000","last_became_peered":"2024-03-11T22:50:32.921834+0000","last_unstale":"2024-03-11T22:50:56.093244+0000","last_undegraded":"2024-03-11T22:50:56.093244+0000","last_fullsized":"2024-03-11T22:50:56.093244+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.029611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951890+0000","last_change":"2024-03-11T22:50:30.921834+0000","last_active":"2024-03-11T22:51:11.951890+0000","last_peered":"2024-03-11T22:51:11.951890+0000","last_clean":"2024-03-11T22:51:11.951890+0000","last_became_active":"2024-03-11T22:50:30.920903+0000","last_became_peered":"2024-03-11T22:50:30.920903+0000","last_unstale":"2024-03-11T22:51:11.951890+0000","last_undegraded":"2024-03-11T22:51:11.951890+0000","last_fullsized":"2024-03-11T22:51:11.951890+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:46:42.395190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094165+0000","last_change":"2024-03-11T22:50:26.900420+0000","last_active":"2024-03-11T22:50:56.094165+0000","last_peered":"2024-03-11T22:50:56.094165+0000","last_clean":"2024-03-11T22:50:56.094165+0000","last_became_active":"2024-03-11T22:50:26.899236+0000","last_became_peered":"2024-03-11T22:50:26.899236+0000","last_unstale":"2024-03-11T22:50:56.094165+0000","last_undegraded":"2024-03-11T22:50:56.094165+0000","last_fullsized":"2024-03-11T22:50:56.094165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:02:43.949862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092528+0000","last_change":"2024-03-11T22:50:28.910125+0000","last_active":"2024-03-11T22:50:56.092528+0000","last_peered":"2024-03-11T22:50:56.092528+0000","last_clean":"2024-03-11T22:50:56.092528+0000","last_became_active":"2024-03-11T22:50:28.909772+0000","last_became_peered":"2024-03-11T22:50:28.909772+0000","last_unstale":"2024-03-11T22:50:56.092528+0000","last_undegraded":"2024-03-11T22:50:56.092528+0000","last_fullsized":"2024-03-11T22:50:56.092528+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:50.991615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096334+0000","last_change":"2024-03-11T22:50:32.923994+0000","last_active":"2024-03-11T22:50:56.096334+0000","last_peered":"2024-03-11T22:50:56.096334+0000","last_clean":"2024-03-11T22:50:56.096334+0000","last_became_active":"2024-03-11T22:50:32.923641+0000","last_became_peered":"2024-03-11T22:50:32.923641+0000","last_unstale":"2024-03-11T22:50:56.096334+0000","last_undegraded":"2024-03-11T22:50:56.096334+0000","last_fullsized":"2024-03-11T22:50:56.096334+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:22:12.491929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093405+0000","last_change":"2024-03-11T22:50:30.922334+0000","last_active":"2024-03-11T22:50:56.093405+0000","last_peered":"2024-03-11T22:50:56.093405+0000","last_clean":"2024-03-11T22:50:56.093405+0000","last_became_active":"2024-03-11T22:50:30.921916+0000","last_became_peered":"2024-03-11T22:50:30.921916+0000","last_unstale":"2024-03-11T22:50:56.093405+0000","last_undegraded":"2024-03-11T22:50:56.093405+0000","last_fullsized":"2024-03-11T22:50:56.093405+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:30.676606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094383+0000","last_change":"2024-03-11T22:50:26.901591+0000","last_active":"2024-03-11T22:50:56.094383+0000","last_peered":"2024-03-11T22:50:56.094383+0000","last_clean":"2024-03-11T22:50:56.094383+0000","last_became_active":"2024-03-11T22:50:26.900637+0000","last_became_peered":"2024-03-11T22:50:26.900637+0000","last_unstale":"2024-03-11T22:50:56.094383+0000","last_undegraded":"2024-03-11T22:50:56.094383+0000","last_fullsized":"2024-03-11T22:50:56.094383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:25:27.033569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092765+0000","last_change":"2024-03-11T22:50:28.908682+0000","last_active":"2024-03-11T22:50:56.092765+0000","last_peered":"2024-03-11T22:50:56.092765+0000","last_clean":"2024-03-11T22:50:56.092765+0000","last_became_active":"2024-03-11T22:50:28.908408+0000","last_became_peered":"2024-03-11T22:50:28.908408+0000","last_unstale":"2024-03-11T22:50:56.092765+0000","last_undegraded":"2024-03-11T22:50:56.092765+0000","last_fullsized":"2024-03-11T22:50:56.092765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:19.041424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095257+0000","last_change":"2024-03-11T22:50:32.917075+0000","last_active":"2024-03-11T22:50:56.095257+0000","last_peered":"2024-03-11T22:50:56.095257+0000","last_clean":"2024-03-11T22:50:56.095257+0000","last_became_active":"2024-03-11T22:50:32.916756+0000","last_became_peered":"2024-03-11T22:50:32.916756+0000","last_unstale":"2024-03-11T22:50:56.095257+0000","last_undegraded":"2024-03-11T22:50:56.095257+0000","last_fullsized":"2024-03-11T22:50:56.095257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:47.892516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092588+0000","last_change":"2024-03-11T22:50:30.912428+0000","last_active":"2024-03-11T22:50:56.092588+0000","last_peered":"2024-03-11T22:50:56.092588+0000","last_clean":"2024-03-11T22:50:56.092588+0000","last_became_active":"2024-03-11T22:50:30.912052+0000","last_became_peered":"2024-03-11T22:50:30.912052+0000","last_unstale":"2024-03-11T22:50:56.092588+0000","last_undegraded":"2024-03-11T22:50:56.092588+0000","last_fullsized":"2024-03-11T22:50:56.092588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:22:10.011551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096973+0000","last_change":"2024-03-11T22:50:26.902823+0000","last_active":"2024-03-11T22:50:56.096973+0000","last_peered":"2024-03-11T22:50:56.096973+0000","last_clean":"2024-03-11T22:50:56.096973+0000","last_became_active":"2024-03-11T22:50:26.902410+0000","last_became_peered":"2024-03-11T22:50:26.902410+0000","last_unstale":"2024-03-11T22:50:56.096973+0000","last_undegraded":"2024-03-11T22:50:56.096973+0000","last_fullsized":"2024-03-11T22:50:56.096973+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:55:27.791788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096368+0000","last_change":"2024-03-11T22:50:28.911574+0000","last_active":"2024-03-11T22:50:56.096368+0000","last_peered":"2024-03-11T22:50:56.096368+0000","last_clean":"2024-03-11T22:50:56.096368+0000","last_became_active":"2024-03-11T22:50:28.910483+0000","last_became_peered":"2024-03-11T22:50:28.910483+0000","last_unstale":"2024-03-11T22:50:56.096368+0000","last_undegraded":"2024-03-11T22:50:56.096368+0000","last_fullsized":"2024-03-11T22:50:56.096368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:01.197107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093784+0000","last_change":"2024-03-11T22:50:32.917035+0000","last_active":"2024-03-11T22:50:56.093784+0000","last_peered":"2024-03-11T22:50:56.093784+0000","last_clean":"2024-03-11T22:50:56.093784+0000","last_became_active":"2024-03-11T22:50:32.916689+0000","last_became_peered":"2024-03-11T22:50:32.916689+0000","last_unstale":"2024-03-11T22:50:56.093784+0000","last_undegraded":"2024-03-11T22:50:56.093784+0000","last_fullsized":"2024-03-11T22:50:56.093784+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:07:25.780960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092914+0000","last_change":"2024-03-11T22:50:30.921519+0000","last_active":"2024-03-11T22:50:56.092914+0000","last_peered":"2024-03-11T22:50:56.092914+0000","last_clean":"2024-03-11T22:50:56.092914+0000","last_became_active":"2024-03-11T22:50:30.921083+0000","last_became_peered":"2024-03-11T22:50:30.921083+0000","last_unstale":"2024-03-11T22:50:56.092914+0000","last_undegraded":"2024-03-11T22:50:56.092914+0000","last_fullsized":"2024-03-11T22:50:56.092914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:49:52.461231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096524+0000","last_change":"2024-03-11T22:50:26.904050+0000","last_active":"2024-03-11T22:50:56.096524+0000","last_peered":"2024-03-11T22:50:56.096524+0000","last_clean":"2024-03-11T22:50:56.096524+0000","last_became_active":"2024-03-11T22:50:26.903615+0000","last_became_peered":"2024-03-11T22:50:26.903615+0000","last_unstale":"2024-03-11T22:50:56.096524+0000","last_undegraded":"2024-03-11T22:50:56.096524+0000","last_fullsized":"2024-03-11T22:50:56.096524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:31:07.661888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095631+0000","last_change":"2024-03-11T22:50:28.910648+0000","last_active":"2024-03-11T22:50:56.095631+0000","last_peered":"2024-03-11T22:50:56.095631+0000","last_clean":"2024-03-11T22:50:56.095631+0000","last_became_active":"2024-03-11T22:50:28.910032+0000","last_became_peered":"2024-03-11T22:50:28.910032+0000","last_unstale":"2024-03-11T22:50:56.095631+0000","last_undegraded":"2024-03-11T22:50:56.095631+0000","last_fullsized":"2024-03-11T22:50:56.095631+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:17:36.534219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094500+0000","last_change":"2024-03-11T22:50:32.919497+0000","last_active":"2024-03-11T22:50:56.094500+0000","last_peered":"2024-03-11T22:50:56.094500+0000","last_clean":"2024-03-11T22:50:56.094500+0000","last_became_active":"2024-03-11T22:50:32.918980+0000","last_became_peered":"2024-03-11T22:50:32.918980+0000","last_unstale":"2024-03-11T22:50:56.094500+0000","last_undegraded":"2024-03-11T22:50:56.094500+0000","last_fullsized":"2024-03-11T22:50:56.094500+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:57:23.223957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096217+0000","last_change":"2024-03-11T22:50:30.919841+0000","last_active":"2024-03-11T22:50:56.096217+0000","last_peered":"2024-03-11T22:50:56.096217+0000","last_clean":"2024-03-11T22:50:56.096217+0000","last_became_active":"2024-03-11T22:50:30.919160+0000","last_became_peered":"2024-03-11T22:50:30.919160+0000","last_unstale":"2024-03-11T22:50:56.096217+0000","last_undegraded":"2024-03-11T22:50:56.096217+0000","last_fullsized":"2024-03-11T22:50:56.096217+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:33.341312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097185+0000","last_change":"2024-03-11T22:50:26.903671+0000","last_active":"2024-03-11T22:50:56.097185+0000","last_peered":"2024-03-11T22:50:56.097185+0000","last_clean":"2024-03-11T22:50:56.097185+0000","last_became_active":"2024-03-11T22:50:26.901916+0000","last_became_peered":"2024-03-11T22:50:26.901916+0000","last_unstale":"2024-03-11T22:50:56.097185+0000","last_undegraded":"2024-03-11T22:50:56.097185+0000","last_fullsized":"2024-03-11T22:50:56.097185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:46:11.568818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095069+0000","last_change":"2024-03-11T22:50:28.911686+0000","last_active":"2024-03-11T22:50:56.095069+0000","last_peered":"2024-03-11T22:50:56.095069+0000","last_clean":"2024-03-11T22:50:56.095069+0000","last_became_active":"2024-03-11T22:50:28.910611+0000","last_became_peered":"2024-03-11T22:50:28.910611+0000","last_unstale":"2024-03-11T22:50:56.095069+0000","last_undegraded":"2024-03-11T22:50:56.095069+0000","last_fullsized":"2024-03-11T22:50:56.095069+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:27:55.181948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094464+0000","last_change":"2024-03-11T22:50:32.919512+0000","last_active":"2024-03-11T22:50:56.094464+0000","last_peered":"2024-03-11T22:50:56.094464+0000","last_clean":"2024-03-11T22:50:56.094464+0000","last_became_active":"2024-03-11T22:50:32.918973+0000","last_became_peered":"2024-03-11T22:50:32.918973+0000","last_unstale":"2024-03-11T22:50:56.094464+0000","last_undegraded":"2024-03-11T22:50:56.094464+0000","last_fullsized":"2024-03-11T22:50:56.094464+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:25.237547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092861+0000","last_change":"2024-03-11T22:50:30.914227+0000","last_active":"2024-03-11T22:50:56.092861+0000","last_peered":"2024-03-11T22:50:56.092861+0000","last_clean":"2024-03-11T22:50:56.092861+0000","last_became_active":"2024-03-11T22:50:30.913851+0000","last_became_peered":"2024-03-11T22:50:30.913851+0000","last_unstale":"2024-03-11T22:50:56.092861+0000","last_undegraded":"2024-03-11T22:50:56.092861+0000","last_fullsized":"2024-03-11T22:50:56.092861+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:17.633666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094843+0000","last_change":"2024-03-11T22:50:26.903625+0000","last_active":"2024-03-11T22:50:56.094843+0000","last_peered":"2024-03-11T22:50:56.094843+0000","last_clean":"2024-03-11T22:50:56.094843+0000","last_became_active":"2024-03-11T22:50:26.903206+0000","last_became_peered":"2024-03-11T22:50:26.903206+0000","last_unstale":"2024-03-11T22:50:56.094843+0000","last_undegraded":"2024-03-11T22:50:56.094843+0000","last_fullsized":"2024-03-11T22:50:56.094843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:03:27.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096413+0000","last_change":"2024-03-11T22:50:28.913328+0000","last_active":"2024-03-11T22:50:56.096413+0000","last_peered":"2024-03-11T22:50:56.096413+0000","last_clean":"2024-03-11T22:50:56.096413+0000","last_became_active":"2024-03-11T22:50:28.912923+0000","last_became_peered":"2024-03-11T22:50:28.912923+0000","last_unstale":"2024-03-11T22:50:56.096413+0000","last_undegraded":"2024-03-11T22:50:56.096413+0000","last_fullsized":"2024-03-11T22:50:56.096413+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:32:54.273252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093941+0000","last_change":"2024-03-11T22:50:32.921655+0000","last_active":"2024-03-11T22:50:56.093941+0000","last_peered":"2024-03-11T22:50:56.093941+0000","last_clean":"2024-03-11T22:50:56.093941+0000","last_became_active":"2024-03-11T22:50:32.920982+0000","last_became_peered":"2024-03-11T22:50:32.920982+0000","last_unstale":"2024-03-11T22:50:56.093941+0000","last_undegraded":"2024-03-11T22:50:56.093941+0000","last_fullsized":"2024-03-11T22:50:56.093941+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:31:45.726492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951853+0000","last_change":"2024-03-11T22:50:30.921133+0000","last_active":"2024-03-11T22:51:11.951853+0000","last_peered":"2024-03-11T22:51:11.951853+0000","last_clean":"2024-03-11T22:51:11.951853+0000","last_became_active":"2024-03-11T22:50:30.920504+0000","last_became_peered":"2024-03-11T22:50:30.920504+0000","last_unstale":"2024-03-11T22:51:11.951853+0000","last_undegraded":"2024-03-11T22:51:11.951853+0000","last_fullsized":"2024-03-11T22:51:11.951853+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:00:31.672875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095045+0000","last_change":"2024-03-11T22:50:26.901345+0000","last_active":"2024-03-11T22:50:56.095045+0000","last_peered":"2024-03-11T22:50:56.095045+0000","last_clean":"2024-03-11T22:50:56.095045+0000","last_became_active":"2024-03-11T22:50:26.900653+0000","last_became_peered":"2024-03-11T22:50:26.900653+0000","last_unstale":"2024-03-11T22:50:56.095045+0000","last_undegraded":"2024-03-11T22:50:56.095045+0000","last_fullsized":"2024-03-11T22:50:56.095045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:17:48.520316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093392+0000","last_change":"2024-03-11T22:50:28.910131+0000","last_active":"2024-03-11T22:50:56.093392+0000","last_peered":"2024-03-11T22:50:56.093392+0000","last_clean":"2024-03-11T22:50:56.093392+0000","last_became_active":"2024-03-11T22:50:28.909751+0000","last_became_peered":"2024-03-11T22:50:28.909751+0000","last_unstale":"2024-03-11T22:50:56.093392+0000","last_undegraded":"2024-03-11T22:50:56.093392+0000","last_fullsized":"2024-03-11T22:50:56.093392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:21.451987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096157+0000","last_change":"2024-03-11T22:50:32.920843+0000","last_active":"2024-03-11T22:50:56.096157+0000","last_peered":"2024-03-11T22:50:56.096157+0000","last_clean":"2024-03-11T22:50:56.096157+0000","last_became_active":"2024-03-11T22:50:32.920329+0000","last_became_peered":"2024-03-11T22:50:32.920329+0000","last_unstale":"2024-03-11T22:50:56.096157+0000","last_undegraded":"2024-03-11T22:50:56.096157+0000","last_fullsized":"2024-03-11T22:50:56.096157+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:19:40.858880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951921+0000","last_change":"2024-03-11T22:50:30.920943+0000","last_active":"2024-03-11T22:51:11.951921+0000","last_peered":"2024-03-11T22:51:11.951921+0000","last_clean":"2024-03-11T22:51:11.951921+0000","last_became_active":"2024-03-11T22:50:30.920647+0000","last_became_peered":"2024-03-11T22:50:30.920647+0000","last_unstale":"2024-03-11T22:51:11.951921+0000","last_undegraded":"2024-03-11T22:51:11.951921+0000","last_fullsized":"2024-03-11T22:51:11.951921+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:00:48.958373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094762+0000","last_change":"2024-03-11T22:50:26.901176+0000","last_active":"2024-03-11T22:50:56.094762+0000","last_peered":"2024-03-11T22:50:56.094762+0000","last_clean":"2024-03-11T22:50:56.094762+0000","last_became_active":"2024-03-11T22:50:26.900572+0000","last_became_peered":"2024-03-11T22:50:26.900572+0000","last_unstale":"2024-03-11T22:50:56.094762+0000","last_undegraded":"2024-03-11T22:50:56.094762+0000","last_fullsized":"2024-03-11T22:50:56.094762+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:48:18.275141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096443+0000","last_change":"2024-03-11T22:50:28.910277+0000","last_active":"2024-03-11T22:50:56.096443+0000","last_peered":"2024-03-11T22:50:56.096443+0000","last_clean":"2024-03-11T22:50:56.096443+0000","last_became_active":"2024-03-11T22:50:28.909734+0000","last_became_peered":"2024-03-11T22:50:28.909734+0000","last_unstale":"2024-03-11T22:50:56.096443+0000","last_undegraded":"2024-03-11T22:50:56.096443+0000","last_fullsized":"2024-03-11T22:50:56.096443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:45.990785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094376+0000","last_change":"2024-03-11T22:50:32.924198+0000","last_active":"2024-03-11T22:50:56.094376+0000","last_peered":"2024-03-11T22:50:56.094376+0000","last_clean":"2024-03-11T22:50:56.094376+0000","last_became_active":"2024-03-11T22:50:32.923802+0000","last_became_peered":"2024-03-11T22:50:32.923802+0000","last_unstale":"2024-03-11T22:50:56.094376+0000","last_undegraded":"2024-03-11T22:50:56.094376+0000","last_fullsized":"2024-03-11T22:50:56.094376+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:46.861200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094564+0000","last_change":"2024-03-11T22:50:30.921967+0000","last_active":"2024-03-11T22:50:56.094564+0000","last_peered":"2024-03-11T22:50:56.094564+0000","last_clean":"2024-03-11T22:50:56.094564+0000","last_became_active":"2024-03-11T22:50:30.921589+0000","last_became_peered":"2024-03-11T22:50:30.921589+0000","last_unstale":"2024-03-11T22:50:56.094564+0000","last_undegraded":"2024-03-11T22:50:56.094564+0000","last_fullsized":"2024-03-11T22:50:56.094564+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:57:44.858552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093909+0000","last_change":"2024-03-11T22:50:26.904482+0000","last_active":"2024-03-11T22:50:56.093909+0000","last_peered":"2024-03-11T22:50:56.093909+0000","last_clean":"2024-03-11T22:50:56.093909+0000","last_became_active":"2024-03-11T22:50:26.904032+0000","last_became_peered":"2024-03-11T22:50:26.904032+0000","last_unstale":"2024-03-11T22:50:56.093909+0000","last_undegraded":"2024-03-11T22:50:56.093909+0000","last_fullsized":"2024-03-11T22:50:56.093909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:32:39.080934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093623+0000","last_change":"2024-03-11T22:50:28.911457+0000","last_active":"2024-03-11T22:50:56.093623+0000","last_peered":"2024-03-11T22:50:56.093623+0000","last_clean":"2024-03-11T22:50:56.093623+0000","last_became_active":"2024-03-11T22:50:28.909930+0000","last_became_peered":"2024-03-11T22:50:28.909930+0000","last_unstale":"2024-03-11T22:50:56.093623+0000","last_undegraded":"2024-03-11T22:50:56.093623+0000","last_fullsized":"2024-03-11T22:50:56.093623+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:26.804925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094245+0000","last_change":"2024-03-11T22:50:32.920993+0000","last_active":"2024-03-11T22:50:56.094245+0000","last_peered":"2024-03-11T22:50:56.094245+0000","last_clean":"2024-03-11T22:50:56.094245+0000","last_became_active":"2024-03-11T22:50:32.920546+0000","last_became_peered":"2024-03-11T22:50:32.920546+0000","last_unstale":"2024-03-11T22:50:56.094245+0000","last_undegraded":"2024-03-11T22:50:56.094245+0000","last_fullsized":"2024-03-11T22:50:56.094245+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:36:37.571372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093498+0000","last_change":"2024-03-11T22:50:30.921565+0000","last_active":"2024-03-11T22:50:56.093498+0000","last_peered":"2024-03-11T22:50:56.093498+0000","last_clean":"2024-03-11T22:50:56.093498+0000","last_became_active":"2024-03-11T22:50:30.920617+0000","last_became_peered":"2024-03-11T22:50:30.920617+0000","last_unstale":"2024-03-11T22:50:56.093498+0000","last_undegraded":"2024-03-11T22:50:56.093498+0000","last_fullsized":"2024-03-11T22:50:56.093498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:39.490798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094528+0000","last_change":"2024-03-11T22:50:26.901643+0000","last_active":"2024-03-11T22:50:56.094528+0000","last_peered":"2024-03-11T22:50:56.094528+0000","last_clean":"2024-03-11T22:50:56.094528+0000","last_became_active":"2024-03-11T22:50:26.901029+0000","last_became_peered":"2024-03-11T22:50:26.901029+0000","last_unstale":"2024-03-11T22:50:56.094528+0000","last_undegraded":"2024-03-11T22:50:56.094528+0000","last_fullsized":"2024-03-11T22:50:56.094528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:10:11.840759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095375+0000","last_change":"2024-03-11T22:50:28.912593+0000","last_active":"2024-03-11T22:50:56.095375+0000","last_peered":"2024-03-11T22:50:56.095375+0000","last_clean":"2024-03-11T22:50:56.095375+0000","last_became_active":"2024-03-11T22:50:28.912297+0000","last_became_peered":"2024-03-11T22:50:28.912297+0000","last_unstale":"2024-03-11T22:50:56.095375+0000","last_undegraded":"2024-03-11T22:50:56.095375+0000","last_fullsized":"2024-03-11T22:50:56.095375+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:23.501238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095360+0000","last_change":"2024-03-11T22:50:32.917771+0000","last_active":"2024-03-11T22:50:56.095360+0000","last_peered":"2024-03-11T22:50:56.095360+0000","last_clean":"2024-03-11T22:50:56.095360+0000","last_became_active":"2024-03-11T22:50:32.917545+0000","last_became_peered":"2024-03-11T22:50:32.917545+0000","last_unstale":"2024-03-11T22:50:56.095360+0000","last_undegraded":"2024-03-11T22:50:56.095360+0000","last_fullsized":"2024-03-11T22:50:56.095360+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:34.091143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097195+0000","last_change":"2024-03-11T22:50:26.903500+0000","last_active":"2024-03-11T22:50:56.097195+0000","last_peered":"2024-03-11T22:50:56.097195+0000","last_clean":"2024-03-11T22:50:56.097195+0000","last_became_active":"2024-03-11T22:50:26.902331+0000","last_became_peered":"2024-03-11T22:50:26.902331+0000","last_unstale":"2024-03-11T22:50:56.097195+0000","last_undegraded":"2024-03-11T22:50:56.097195+0000","last_fullsized":"2024-03-11T22:50:56.097195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:40.049766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097060+0000","last_change":"2024-03-11T22:50:28.910021+0000","last_active":"2024-03-11T22:50:56.097060+0000","last_peered":"2024-03-11T22:50:56.097060+0000","last_clean":"2024-03-11T22:50:56.097060+0000","last_became_active":"2024-03-11T22:50:28.909735+0000","last_became_peered":"2024-03-11T22:50:28.909735+0000","last_unstale":"2024-03-11T22:50:56.097060+0000","last_undegraded":"2024-03-11T22:50:56.097060+0000","last_fullsized":"2024-03-11T22:50:56.097060+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:35:50.874827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951893+0000","last_change":"2024-03-11T22:50:30.919739+0000","last_active":"2024-03-11T22:51:11.951893+0000","last_peered":"2024-03-11T22:51:11.951893+0000","last_clean":"2024-03-11T22:51:11.951893+0000","last_became_active":"2024-03-11T22:50:30.919276+0000","last_became_peered":"2024-03-11T22:50:30.919276+0000","last_unstale":"2024-03-11T22:51:11.951893+0000","last_undegraded":"2024-03-11T22:51:11.951893+0000","last_fullsized":"2024-03-11T22:51:11.951893+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:22:54.554795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093024+0000","last_change":"2024-03-11T22:50:32.918992+0000","last_active":"2024-03-11T22:50:56.093024+0000","last_peered":"2024-03-11T22:50:56.093024+0000","last_clean":"2024-03-11T22:50:56.093024+0000","last_became_active":"2024-03-11T22:50:32.917545+0000","last_became_peered":"2024-03-11T22:50:32.917545+0000","last_unstale":"2024-03-11T22:50:56.093024+0000","last_undegraded":"2024-03-11T22:50:56.093024+0000","last_fullsized":"2024-03-11T22:50:56.093024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:12:58.613786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094164+0000","last_change":"2024-03-11T22:50:26.899067+0000","last_active":"2024-03-11T22:50:56.094164+0000","last_peered":"2024-03-11T22:50:56.094164+0000","last_clean":"2024-03-11T22:50:56.094164+0000","last_became_active":"2024-03-11T22:50:26.898249+0000","last_became_peered":"2024-03-11T22:50:26.898249+0000","last_unstale":"2024-03-11T22:50:56.094164+0000","last_undegraded":"2024-03-11T22:50:56.094164+0000","last_fullsized":"2024-03-11T22:50:56.094164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:27.893635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951762+0000","last_change":"2024-03-11T22:50:28.908954+0000","last_active":"2024-03-11T22:51:11.951762+0000","last_peered":"2024-03-11T22:51:11.951762+0000","last_clean":"2024-03-11T22:51:11.951762+0000","last_became_active":"2024-03-11T22:50:28.908195+0000","last_became_peered":"2024-03-11T22:50:28.908195+0000","last_unstale":"2024-03-11T22:51:11.951762+0000","last_undegraded":"2024-03-11T22:51:11.951762+0000","last_fullsized":"2024-03-11T22:51:11.951762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:48.956413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094599+0000","last_change":"2024-03-11T22:50:30.920139+0000","last_active":"2024-03-11T22:50:56.094599+0000","last_peered":"2024-03-11T22:50:56.094599+0000","last_clean":"2024-03-11T22:50:56.094599+0000","last_became_active":"2024-03-11T22:50:30.919701+0000","last_became_peered":"2024-03-11T22:50:30.919701+0000","last_unstale":"2024-03-11T22:50:56.094599+0000","last_undegraded":"2024-03-11T22:50:56.094599+0000","last_fullsized":"2024-03-11T22:50:56.094599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:18:39.520668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094385+0000","last_change":"2024-03-11T22:50:32.921282+0000","last_active":"2024-03-11T22:50:56.094385+0000","last_peered":"2024-03-11T22:50:56.094385+0000","last_clean":"2024-03-11T22:50:56.094385+0000","last_became_active":"2024-03-11T22:50:32.920902+0000","last_became_peered":"2024-03-11T22:50:32.920902+0000","last_unstale":"2024-03-11T22:50:56.094385+0000","last_undegraded":"2024-03-11T22:50:56.094385+0000","last_fullsized":"2024-03-11T22:50:56.094385+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:11:32.247011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094761+0000","last_change":"2024-03-11T22:50:26.902811+0000","last_active":"2024-03-11T22:50:56.094761+0000","last_peered":"2024-03-11T22:50:56.094761+0000","last_clean":"2024-03-11T22:50:56.094761+0000","last_became_active":"2024-03-11T22:50:26.902371+0000","last_became_peered":"2024-03-11T22:50:26.902371+0000","last_unstale":"2024-03-11T22:50:56.094761+0000","last_undegraded":"2024-03-11T22:50:56.094761+0000","last_fullsized":"2024-03-11T22:50:56.094761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:03:23.892861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096848+0000","last_change":"2024-03-11T22:50:28.909098+0000","last_active":"2024-03-11T22:50:56.096848+0000","last_peered":"2024-03-11T22:50:56.096848+0000","last_clean":"2024-03-11T22:50:56.096848+0000","last_became_active":"2024-03-11T22:50:28.908451+0000","last_became_peered":"2024-03-11T22:50:28.908451+0000","last_unstale":"2024-03-11T22:50:56.096848+0000","last_undegraded":"2024-03-11T22:50:56.096848+0000","last_fullsized":"2024-03-11T22:50:56.096848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:47.483886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094043+0000","last_change":"2024-03-11T22:50:30.920511+0000","last_active":"2024-03-11T22:50:56.094043+0000","last_peered":"2024-03-11T22:50:56.094043+0000","last_clean":"2024-03-11T22:50:56.094043+0000","last_became_active":"2024-03-11T22:50:30.920186+0000","last_became_peered":"2024-03-11T22:50:30.920186+0000","last_unstale":"2024-03-11T22:50:56.094043+0000","last_undegraded":"2024-03-11T22:50:56.094043+0000","last_fullsized":"2024-03-11T22:50:56.094043+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:47.625083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094375+0000","last_change":"2024-03-11T22:50:32.924799+0000","last_active":"2024-03-11T22:50:56.094375+0000","last_peered":"2024-03-11T22:50:56.094375+0000","last_clean":"2024-03-11T22:50:56.094375+0000","last_became_active":"2024-03-11T22:50:32.924511+0000","last_became_peered":"2024-03-11T22:50:32.924511+0000","last_unstale":"2024-03-11T22:50:56.094375+0000","last_undegraded":"2024-03-11T22:50:56.094375+0000","last_fullsized":"2024-03-11T22:50:56.094375+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:55.781975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094145+0000","last_change":"2024-03-11T22:50:28.912915+0000","last_active":"2024-03-11T22:50:56.094145+0000","last_peered":"2024-03-11T22:50:56.094145+0000","last_clean":"2024-03-11T22:50:56.094145+0000","last_became_active":"2024-03-11T22:50:28.912648+0000","last_became_peered":"2024-03-11T22:50:28.912648+0000","last_unstale":"2024-03-11T22:50:56.094145+0000","last_undegraded":"2024-03-11T22:50:56.094145+0000","last_fullsized":"2024-03-11T22:50:56.094145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:11:22.591244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094010+0000","last_change":"2024-03-11T22:50:26.900122+0000","last_active":"2024-03-11T22:50:56.094010+0000","last_peered":"2024-03-11T22:50:56.094010+0000","last_clean":"2024-03-11T22:50:56.094010+0000","last_became_active":"2024-03-11T22:50:26.899230+0000","last_became_peered":"2024-03-11T22:50:26.899230+0000","last_unstale":"2024-03-11T22:50:56.094010+0000","last_undegraded":"2024-03-11T22:50:56.094010+0000","last_fullsized":"2024-03-11T22:50:56.094010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:27:04.428241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095594+0000","last_change":"2024-03-11T22:50:30.920899+0000","last_active":"2024-03-11T22:50:56.095594+0000","last_peered":"2024-03-11T22:50:56.095594+0000","last_clean":"2024-03-11T22:50:56.095594+0000","last_became_active":"2024-03-11T22:50:30.920259+0000","last_became_peered":"2024-03-11T22:50:30.920259+0000","last_unstale":"2024-03-11T22:50:56.095594+0000","last_undegraded":"2024-03-11T22:50:56.095594+0000","last_fullsized":"2024-03-11T22:50:56.095594+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:32:38.776158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095267+0000","last_change":"2024-03-11T22:50:32.919764+0000","last_active":"2024-03-11T22:50:56.095267+0000","last_peered":"2024-03-11T22:50:56.095267+0000","last_clean":"2024-03-11T22:50:56.095267+0000","last_became_active":"2024-03-11T22:50:32.919442+0000","last_became_peered":"2024-03-11T22:50:32.919442+0000","last_unstale":"2024-03-11T22:50:56.095267+0000","last_undegraded":"2024-03-11T22:50:56.095267+0000","last_fullsized":"2024-03-11T22:50:56.095267+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:52:02.533788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093141+0000","last_change":"2024-03-11T22:50:28.908575+0000","last_active":"2024-03-11T22:50:56.093141+0000","last_peered":"2024-03-11T22:50:56.093141+0000","last_clean":"2024-03-11T22:50:56.093141+0000","last_became_active":"2024-03-11T22:50:28.908093+0000","last_became_peered":"2024-03-11T22:50:28.908093+0000","last_unstale":"2024-03-11T22:50:56.093141+0000","last_undegraded":"2024-03-11T22:50:56.093141+0000","last_fullsized":"2024-03-11T22:50:56.093141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:54:13.708444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096581+0000","last_change":"2024-03-11T22:50:26.902961+0000","last_active":"2024-03-11T22:50:56.096581+0000","last_peered":"2024-03-11T22:50:56.096581+0000","last_clean":"2024-03-11T22:50:56.096581+0000","last_became_active":"2024-03-11T22:50:26.902420+0000","last_became_peered":"2024-03-11T22:50:26.902420+0000","last_unstale":"2024-03-11T22:50:56.096581+0000","last_undegraded":"2024-03-11T22:50:56.096581+0000","last_fullsized":"2024-03-11T22:50:56.096581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:47:19.560282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:58.203664+0000","last_change":"2024-03-11T22:50:15.659487+0000","last_active":"2024-03-11T22:50:58.203664+0000","last_peered":"2024-03-11T22:50:58.203664+0000","last_clean":"2024-03-11T22:50:58.203664+0000","last_became_active":"2024-03-11T22:50:15.659173+0000","last_became_peered":"2024-03-11T22:50:15.659173+0000","last_unstale":"2024-03-11T22:50:58.203664+0000","last_undegraded":"2024-03-11T22:50:58.203664+0000","last_fullsized":"2024-03-11T22:50:58.203664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:14.645525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:14.645525+0000","last_clean_scrub_stamp":"2024-03-11T22:50:14.645525+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:30.563534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094642+0000","last_change":"2024-03-11T22:50:30.916891+0000","last_active":"2024-03-11T22:50:56.094642+0000","last_peered":"2024-03-11T22:50:56.094642+0000","last_clean":"2024-03-11T22:50:56.094642+0000","last_became_active":"2024-03-11T22:50:30.915600+0000","last_became_peered":"2024-03-11T22:50:30.915600+0000","last_unstale":"2024-03-11T22:50:56.094642+0000","last_undegraded":"2024-03-11T22:50:56.094642+0000","last_fullsized":"2024-03-11T22:50:56.094642+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:21:45.576873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096512+0000","last_change":"2024-03-11T22:50:32.920993+0000","last_active":"2024-03-11T22:50:56.096512+0000","last_peered":"2024-03-11T22:50:56.096512+0000","last_clean":"2024-03-11T22:50:56.096512+0000","last_became_active":"2024-03-11T22:50:32.920596+0000","last_became_peered":"2024-03-11T22:50:32.920596+0000","last_unstale":"2024-03-11T22:50:56.096512+0000","last_undegraded":"2024-03-11T22:50:56.096512+0000","last_fullsized":"2024-03-11T22:50:56.096512+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:32:41.364871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095838+0000","last_change":"2024-03-11T22:50:28.908640+0000","last_active":"2024-03-11T22:50:56.095838+0000","last_peered":"2024-03-11T22:50:56.095838+0000","last_clean":"2024-03-11T22:50:56.095838+0000","last_became_active":"2024-03-11T22:50:28.908295+0000","last_became_peered":"2024-03-11T22:50:28.908295+0000","last_unstale":"2024-03-11T22:50:56.095838+0000","last_undegraded":"2024-03-11T22:50:56.095838+0000","last_fullsized":"2024-03-11T22:50:56.095838+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:08.294780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093576+0000","last_change":"2024-03-11T22:50:26.904372+0000","last_active":"2024-03-11T22:50:56.093576+0000","last_peered":"2024-03-11T22:50:56.093576+0000","last_clean":"2024-03-11T22:50:56.093576+0000","last_became_active":"2024-03-11T22:50:26.903883+0000","last_became_peered":"2024-03-11T22:50:26.903883+0000","last_unstale":"2024-03-11T22:50:56.093576+0000","last_undegraded":"2024-03-11T22:50:56.093576+0000","last_fullsized":"2024-03-11T22:50:56.093576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:36:59.664071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096255+0000","last_change":"2024-03-11T22:50:30.919794+0000","last_active":"2024-03-11T22:50:56.096255+0000","last_peered":"2024-03-11T22:50:56.096255+0000","last_clean":"2024-03-11T22:50:56.096255+0000","last_became_active":"2024-03-11T22:50:30.919455+0000","last_became_peered":"2024-03-11T22:50:30.919455+0000","last_unstale":"2024-03-11T22:50:56.096255+0000","last_undegraded":"2024-03-11T22:50:56.096255+0000","last_fullsized":"2024-03-11T22:50:56.096255+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:33.535853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094531+0000","last_change":"2024-03-11T22:50:32.919924+0000","last_active":"2024-03-11T22:50:56.094531+0000","last_peered":"2024-03-11T22:50:56.094531+0000","last_clean":"2024-03-11T22:50:56.094531+0000","last_became_active":"2024-03-11T22:50:32.919494+0000","last_became_peered":"2024-03-11T22:50:32.919494+0000","last_unstale":"2024-03-11T22:50:56.094531+0000","last_undegraded":"2024-03-11T22:50:56.094531+0000","last_fullsized":"2024-03-11T22:50:56.094531+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:42:14.537886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096465+0000","last_change":"2024-03-11T22:50:26.903712+0000","last_active":"2024-03-11T22:50:56.096465+0000","last_peered":"2024-03-11T22:50:56.096465+0000","last_clean":"2024-03-11T22:50:56.096465+0000","last_became_active":"2024-03-11T22:50:26.902432+0000","last_became_peered":"2024-03-11T22:50:26.902432+0000","last_unstale":"2024-03-11T22:50:56.096465+0000","last_undegraded":"2024-03-11T22:50:56.096465+0000","last_fullsized":"2024-03-11T22:50:56.096465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:06.577403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":83,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952380+0000","last_change":"2024-03-11T22:50:28.909230+0000","last_active":"2024-03-11T22:51:16.952380+0000","last_peered":"2024-03-11T22:51:16.952380+0000","last_clean":"2024-03-11T22:51:16.952380+0000","last_became_active":"2024-03-11T22:50:28.908935+0000","last_became_peered":"2024-03-11T22:50:28.908935+0000","last_unstale":"2024-03-11T22:51:16.952380+0000","last_undegraded":"2024-03-11T22:51:16.952380+0000","last_fullsized":"2024-03-11T22:51:16.952380+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:47:44.057404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951911+0000","last_change":"2024-03-11T22:50:30.921404+0000","last_active":"2024-03-11T22:51:11.951911+0000","last_peered":"2024-03-11T22:51:11.951911+0000","last_clean":"2024-03-11T22:51:11.951911+0000","last_became_active":"2024-03-11T22:50:30.921084+0000","last_became_peered":"2024-03-11T22:50:30.921084+0000","last_unstale":"2024-03-11T22:51:11.951911+0000","last_undegraded":"2024-03-11T22:51:11.951911+0000","last_fullsized":"2024-03-11T22:51:11.951911+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:58:25.723151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093533+0000","last_change":"2024-03-11T22:50:32.920690+0000","last_active":"2024-03-11T22:50:56.093533+0000","last_peered":"2024-03-11T22:50:56.093533+0000","last_clean":"2024-03-11T22:50:56.093533+0000","last_became_active":"2024-03-11T22:50:32.919747+0000","last_became_peered":"2024-03-11T22:50:32.919747+0000","last_unstale":"2024-03-11T22:50:56.093533+0000","last_undegraded":"2024-03-11T22:50:56.093533+0000","last_fullsized":"2024-03-11T22:50:56.093533+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:21:30.334268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094107+0000","last_change":"2024-03-11T22:50:26.900499+0000","last_active":"2024-03-11T22:50:56.094107+0000","last_peered":"2024-03-11T22:50:56.094107+0000","last_clean":"2024-03-11T22:50:56.094107+0000","last_became_active":"2024-03-11T22:50:26.899307+0000","last_became_peered":"2024-03-11T22:50:26.899307+0000","last_unstale":"2024-03-11T22:50:56.094107+0000","last_undegraded":"2024-03-11T22:50:56.094107+0000","last_fullsized":"2024-03-11T22:50:56.094107+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:28:18.008321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095776+0000","last_change":"2024-03-11T22:50:28.910462+0000","last_active":"2024-03-11T22:50:56.095776+0000","last_peered":"2024-03-11T22:50:56.095776+0000","last_clean":"2024-03-11T22:50:56.095776+0000","last_became_active":"2024-03-11T22:50:28.910111+0000","last_became_peered":"2024-03-11T22:50:28.910111+0000","last_unstale":"2024-03-11T22:50:56.095776+0000","last_undegraded":"2024-03-11T22:50:56.095776+0000","last_fullsized":"2024-03-11T22:50:56.095776+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:03:07.224641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952414+0000","last_change":"2024-03-11T22:50:30.921633+0000","last_active":"2024-03-11T22:51:16.952414+0000","last_peered":"2024-03-11T22:51:16.952414+0000","last_clean":"2024-03-11T22:51:16.952414+0000","last_became_active":"2024-03-11T22:50:30.920530+0000","last_became_peered":"2024-03-11T22:50:30.920530+0000","last_unstale":"2024-03-11T22:51:16.952414+0000","last_undegraded":"2024-03-11T22:51:16.952414+0000","last_fullsized":"2024-03-11T22:51:16.952414+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:46:50.900754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095127+0000","last_change":"2024-03-11T22:50:32.919726+0000","last_active":"2024-03-11T22:50:56.095127+0000","last_peered":"2024-03-11T22:50:56.095127+0000","last_clean":"2024-03-11T22:50:56.095127+0000","last_became_active":"2024-03-11T22:50:32.919104+0000","last_became_peered":"2024-03-11T22:50:32.919104+0000","last_unstale":"2024-03-11T22:50:56.095127+0000","last_undegraded":"2024-03-11T22:50:56.095127+0000","last_fullsized":"2024-03-11T22:50:56.095127+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:28.728474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096747+0000","last_change":"2024-03-11T22:50:26.903909+0000","last_active":"2024-03-11T22:50:56.096747+0000","last_peered":"2024-03-11T22:50:56.096747+0000","last_clean":"2024-03-11T22:50:56.096747+0000","last_became_active":"2024-03-11T22:50:26.903301+0000","last_became_peered":"2024-03-11T22:50:26.903301+0000","last_unstale":"2024-03-11T22:50:56.096747+0000","last_undegraded":"2024-03-11T22:50:56.096747+0000","last_fullsized":"2024-03-11T22:50:56.096747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:59:54.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093886+0000","last_change":"2024-03-11T22:50:28.910429+0000","last_active":"2024-03-11T22:50:56.093886+0000","last_peered":"2024-03-11T22:50:56.093886+0000","last_clean":"2024-03-11T22:50:56.093886+0000","last_became_active":"2024-03-11T22:50:28.909904+0000","last_became_peered":"2024-03-11T22:50:28.909904+0000","last_unstale":"2024-03-11T22:50:56.093886+0000","last_undegraded":"2024-03-11T22:50:56.093886+0000","last_fullsized":"2024-03-11T22:50:56.093886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:18.464087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094420+0000","last_change":"2024-03-11T22:50:30.921673+0000","last_active":"2024-03-11T22:50:56.094420+0000","last_peered":"2024-03-11T22:50:56.094420+0000","last_clean":"2024-03-11T22:50:56.094420+0000","last_became_active":"2024-03-11T22:50:30.920291+0000","last_became_peered":"2024-03-11T22:50:30.920291+0000","last_unstale":"2024-03-11T22:50:56.094420+0000","last_undegraded":"2024-03-11T22:50:56.094420+0000","last_fullsized":"2024-03-11T22:50:56.094420+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:53:05.596992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096443+0000","last_change":"2024-03-11T22:50:32.920184+0000","last_active":"2024-03-11T22:50:56.096443+0000","last_peered":"2024-03-11T22:50:56.096443+0000","last_clean":"2024-03-11T22:50:56.096443+0000","last_became_active":"2024-03-11T22:50:32.919815+0000","last_became_peered":"2024-03-11T22:50:32.919815+0000","last_unstale":"2024-03-11T22:50:56.096443+0000","last_undegraded":"2024-03-11T22:50:56.096443+0000","last_fullsized":"2024-03-11T22:50:56.096443+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:27:51.650662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951780+0000","last_change":"2024-03-11T22:50:30.917450+0000","last_active":"2024-03-11T22:51:11.951780+0000","last_peered":"2024-03-11T22:51:11.951780+0000","last_clean":"2024-03-11T22:51:11.951780+0000","last_became_active":"2024-03-11T22:50:30.917059+0000","last_became_peered":"2024-03-11T22:50:30.917059+0000","last_unstale":"2024-03-11T22:51:11.951780+0000","last_undegraded":"2024-03-11T22:51:11.951780+0000","last_fullsized":"2024-03-11T22:51:11.951780+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:24:50.757893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093486+0000","last_change":"2024-03-11T22:50:26.904377+0000","last_active":"2024-03-11T22:50:56.093486+0000","last_peered":"2024-03-11T22:50:56.093486+0000","last_clean":"2024-03-11T22:50:56.093486+0000","last_became_active":"2024-03-11T22:50:26.903973+0000","last_became_peered":"2024-03-11T22:50:26.903973+0000","last_unstale":"2024-03-11T22:50:56.093486+0000","last_undegraded":"2024-03-11T22:50:56.093486+0000","last_fullsized":"2024-03-11T22:50:56.093486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:46.921090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094802+0000","last_change":"2024-03-11T22:50:28.909499+0000","last_active":"2024-03-11T22:50:56.094802+0000","last_peered":"2024-03-11T22:50:56.094802+0000","last_clean":"2024-03-11T22:50:56.094802+0000","last_became_active":"2024-03-11T22:50:28.909054+0000","last_became_peered":"2024-03-11T22:50:28.909054+0000","last_unstale":"2024-03-11T22:50:56.094802+0000","last_undegraded":"2024-03-11T22:50:56.094802+0000","last_fullsized":"2024-03-11T22:50:56.094802+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:10.604872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096187+0000","last_change":"2024-03-11T22:50:32.923767+0000","last_active":"2024-03-11T22:50:56.096187+0000","last_peered":"2024-03-11T22:50:56.096187+0000","last_clean":"2024-03-11T22:50:56.096187+0000","last_became_active":"2024-03-11T22:50:32.923508+0000","last_became_peered":"2024-03-11T22:50:32.923508+0000","last_unstale":"2024-03-11T22:50:56.096187+0000","last_undegraded":"2024-03-11T22:50:56.096187+0000","last_fullsized":"2024-03-11T22:50:56.096187+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:11:20.110632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094568+0000","last_change":"2024-03-11T22:50:30.921573+0000","last_active":"2024-03-11T22:50:56.094568+0000","last_peered":"2024-03-11T22:50:56.094568+0000","last_clean":"2024-03-11T22:50:56.094568+0000","last_became_active":"2024-03-11T22:50:30.920713+0000","last_became_peered":"2024-03-11T22:50:30.920713+0000","last_unstale":"2024-03-11T22:50:56.094568+0000","last_undegraded":"2024-03-11T22:50:56.094568+0000","last_fullsized":"2024-03-11T22:50:56.094568+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:51:16.389553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093839+0000","last_change":"2024-03-11T22:50:26.900505+0000","last_active":"2024-03-11T22:50:56.093839+0000","last_peered":"2024-03-11T22:50:56.093839+0000","last_clean":"2024-03-11T22:50:56.093839+0000","last_became_active":"2024-03-11T22:50:26.900196+0000","last_became_peered":"2024-03-11T22:50:26.900196+0000","last_unstale":"2024-03-11T22:50:56.093839+0000","last_undegraded":"2024-03-11T22:50:56.093839+0000","last_fullsized":"2024-03-11T22:50:56.093839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:16:38.760700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094625+0000","last_change":"2024-03-11T22:50:28.909392+0000","last_active":"2024-03-11T22:50:56.094625+0000","last_peered":"2024-03-11T22:50:56.094625+0000","last_clean":"2024-03-11T22:50:56.094625+0000","last_became_active":"2024-03-11T22:50:28.908984+0000","last_became_peered":"2024-03-11T22:50:28.908984+0000","last_unstale":"2024-03-11T22:50:56.094625+0000","last_undegraded":"2024-03-11T22:50:56.094625+0000","last_fullsized":"2024-03-11T22:50:56.094625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:25:31.458499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095888+0000","last_change":"2024-03-11T22:50:32.919346+0000","last_active":"2024-03-11T22:50:56.095888+0000","last_peered":"2024-03-11T22:50:56.095888+0000","last_clean":"2024-03-11T22:50:56.095888+0000","last_became_active":"2024-03-11T22:50:32.919019+0000","last_became_peered":"2024-03-11T22:50:32.919019+0000","last_unstale":"2024-03-11T22:50:56.095888+0000","last_undegraded":"2024-03-11T22:50:56.095888+0000","last_fullsized":"2024-03-11T22:50:56.095888+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:21.121389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095194+0000","last_change":"2024-03-11T22:50:26.899789+0000","last_active":"2024-03-11T22:50:56.095194+0000","last_peered":"2024-03-11T22:50:56.095194+0000","last_clean":"2024-03-11T22:50:56.095194+0000","last_became_active":"2024-03-11T22:50:26.899464+0000","last_became_peered":"2024-03-11T22:50:26.899464+0000","last_unstale":"2024-03-11T22:50:56.095194+0000","last_undegraded":"2024-03-11T22:50:56.095194+0000","last_fullsized":"2024-03-11T22:50:56.095194+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:23:47.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093743+0000","last_change":"2024-03-11T22:50:28.908081+0000","last_active":"2024-03-11T22:50:56.093743+0000","last_peered":"2024-03-11T22:50:56.093743+0000","last_clean":"2024-03-11T22:50:56.093743+0000","last_became_active":"2024-03-11T22:50:28.907753+0000","last_became_peered":"2024-03-11T22:50:28.907753+0000","last_unstale":"2024-03-11T22:50:56.093743+0000","last_undegraded":"2024-03-11T22:50:56.093743+0000","last_fullsized":"2024-03-11T22:50:56.093743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:21.170870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094617+0000","last_change":"2024-03-11T22:50:30.919101+0000","last_active":"2024-03-11T22:50:56.094617+0000","last_peered":"2024-03-11T22:50:56.094617+0000","last_clean":"2024-03-11T22:50:56.094617+0000","last_became_active":"2024-03-11T22:50:30.918124+0000","last_became_peered":"2024-03-11T22:50:30.918124+0000","last_unstale":"2024-03-11T22:50:56.094617+0000","last_undegraded":"2024-03-11T22:50:56.094617+0000","last_fullsized":"2024-03-11T22:50:56.094617+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:00.364123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092995+0000","last_change":"2024-03-11T22:50:32.920757+0000","last_active":"2024-03-11T22:50:56.092995+0000","last_peered":"2024-03-11T22:50:56.092995+0000","last_clean":"2024-03-11T22:50:56.092995+0000","last_became_active":"2024-03-11T22:50:32.919638+0000","last_became_peered":"2024-03-11T22:50:32.919638+0000","last_unstale":"2024-03-11T22:50:56.092995+0000","last_undegraded":"2024-03-11T22:50:56.092995+0000","last_fullsized":"2024-03-11T22:50:56.092995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:38.621924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:11.951780+0000","last_change":"2024-03-11T22:50:30.919406+0000","last_active":"2024-03-11T22:51:11.951780+0000","last_peered":"2024-03-11T22:51:11.951780+0000","last_clean":"2024-03-11T22:51:11.951780+0000","last_became_active":"2024-03-11T22:50:30.919023+0000","last_became_peered":"2024-03-11T22:50:30.919023+0000","last_unstale":"2024-03-11T22:51:11.951780+0000","last_undegraded":"2024-03-11T22:51:11.951780+0000","last_fullsized":"2024-03-11T22:51:11.951780+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:00:02.577888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095231+0000","last_change":"2024-03-11T22:50:26.899786+0000","last_active":"2024-03-11T22:50:56.095231+0000","last_peered":"2024-03-11T22:50:56.095231+0000","last_clean":"2024-03-11T22:50:56.095231+0000","last_became_active":"2024-03-11T22:50:26.899392+0000","last_became_peered":"2024-03-11T22:50:26.899392+0000","last_unstale":"2024-03-11T22:50:56.095231+0000","last_undegraded":"2024-03-11T22:50:56.095231+0000","last_fullsized":"2024-03-11T22:50:56.095231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:01:28.909407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094393+0000","last_change":"2024-03-11T22:50:28.908445+0000","last_active":"2024-03-11T22:50:56.094393+0000","last_peered":"2024-03-11T22:50:56.094393+0000","last_clean":"2024-03-11T22:50:56.094393+0000","last_became_active":"2024-03-11T22:50:28.907844+0000","last_became_peered":"2024-03-11T22:50:28.907844+0000","last_unstale":"2024-03-11T22:50:56.094393+0000","last_undegraded":"2024-03-11T22:50:56.094393+0000","last_fullsized":"2024-03-11T22:50:56.094393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:32:31.167181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094578+0000","last_change":"2024-03-11T22:50:32.919282+0000","last_active":"2024-03-11T22:50:56.094578+0000","last_peered":"2024-03-11T22:50:56.094578+0000","last_clean":"2024-03-11T22:50:56.094578+0000","last_became_active":"2024-03-11T22:50:32.918901+0000","last_became_peered":"2024-03-11T22:50:32.918901+0000","last_unstale":"2024-03-11T22:50:56.094578+0000","last_undegraded":"2024-03-11T22:50:56.094578+0000","last_fullsized":"2024-03-11T22:50:56.094578+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:39:58.417112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095748+0000","last_change":"2024-03-11T22:50:26.902721+0000","last_active":"2024-03-11T22:50:56.095748+0000","last_peered":"2024-03-11T22:50:56.095748+0000","last_clean":"2024-03-11T22:50:56.095748+0000","last_became_active":"2024-03-11T22:50:26.901836+0000","last_became_peered":"2024-03-11T22:50:26.901836+0000","last_unstale":"2024-03-11T22:50:56.095748+0000","last_undegraded":"2024-03-11T22:50:56.095748+0000","last_fullsized":"2024-03-11T22:50:56.095748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:44:17.145868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094797+0000","last_change":"2024-03-11T22:50:28.909865+0000","last_active":"2024-03-11T22:50:56.094797+0000","last_peered":"2024-03-11T22:50:56.094797+0000","last_clean":"2024-03-11T22:50:56.094797+0000","last_became_active":"2024-03-11T22:50:28.908723+0000","last_became_peered":"2024-03-11T22:50:28.908723+0000","last_unstale":"2024-03-11T22:50:56.094797+0000","last_undegraded":"2024-03-11T22:50:56.094797+0000","last_fullsized":"2024-03-11T22:50:56.094797+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:22:47.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093886+0000","last_change":"2024-03-11T22:50:30.912723+0000","last_active":"2024-03-11T22:50:56.093886+0000","last_peered":"2024-03-11T22:50:56.093886+0000","last_clean":"2024-03-11T22:50:56.093886+0000","last_became_active":"2024-03-11T22:50:30.912299+0000","last_became_peered":"2024-03-11T22:50:30.912299+0000","last_unstale":"2024-03-11T22:50:56.093886+0000","last_undegraded":"2024-03-11T22:50:56.093886+0000","last_fullsized":"2024-03-11T22:50:56.093886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:57.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092438+0000","last_change":"2024-03-11T22:50:32.918484+0000","last_active":"2024-03-11T22:50:56.092438+0000","last_peered":"2024-03-11T22:50:56.092438+0000","last_clean":"2024-03-11T22:50:56.092438+0000","last_became_active":"2024-03-11T22:50:32.918096+0000","last_became_peered":"2024-03-11T22:50:32.918096+0000","last_unstale":"2024-03-11T22:50:56.092438+0000","last_undegraded":"2024-03-11T22:50:56.092438+0000","last_fullsized":"2024-03-11T22:50:56.092438+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:35:45.888334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096163+0000","last_change":"2024-03-11T22:50:26.903957+0000","last_active":"2024-03-11T22:50:56.096163+0000","last_peered":"2024-03-11T22:50:56.096163+0000","last_clean":"2024-03-11T22:50:56.096163+0000","last_became_active":"2024-03-11T22:50:26.903316+0000","last_became_peered":"2024-03-11T22:50:26.903316+0000","last_unstale":"2024-03-11T22:50:56.096163+0000","last_undegraded":"2024-03-11T22:50:56.096163+0000","last_fullsized":"2024-03-11T22:50:56.096163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:36:42.965662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094931+0000","last_change":"2024-03-11T22:50:28.911567+0000","last_active":"2024-03-11T22:50:56.094931+0000","last_peered":"2024-03-11T22:50:56.094931+0000","last_clean":"2024-03-11T22:50:56.094931+0000","last_became_active":"2024-03-11T22:50:28.910618+0000","last_became_peered":"2024-03-11T22:50:28.910618+0000","last_unstale":"2024-03-11T22:50:56.094931+0000","last_undegraded":"2024-03-11T22:50:56.094931+0000","last_fullsized":"2024-03-11T22:50:56.094931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:38:09.997753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093132+0000","last_change":"2024-03-11T22:50:30.914252+0000","last_active":"2024-03-11T22:50:56.093132+0000","last_peered":"2024-03-11T22:50:56.093132+0000","last_clean":"2024-03-11T22:50:56.093132+0000","last_became_active":"2024-03-11T22:50:30.913872+0000","last_became_peered":"2024-03-11T22:50:30.913872+0000","last_unstale":"2024-03-11T22:50:56.093132+0000","last_undegraded":"2024-03-11T22:50:56.093132+0000","last_fullsized":"2024-03-11T22:50:56.093132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:24:15.526871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093392+0000","last_change":"2024-03-11T22:50:32.919850+0000","last_active":"2024-03-11T22:50:56.093392+0000","last_peered":"2024-03-11T22:50:56.093392+0000","last_clean":"2024-03-11T22:50:56.093392+0000","last_became_active":"2024-03-11T22:50:32.919493+0000","last_became_peered":"2024-03-11T22:50:32.919493+0000","last_unstale":"2024-03-11T22:50:56.093392+0000","last_undegraded":"2024-03-11T22:50:56.093392+0000","last_fullsized":"2024-03-11T22:50:56.093392+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:27:41.693416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093787+0000","last_change":"2024-03-11T22:50:26.900752+0000","last_active":"2024-03-11T22:50:56.093787+0000","last_peered":"2024-03-11T22:50:56.093787+0000","last_clean":"2024-03-11T22:50:56.093787+0000","last_became_active":"2024-03-11T22:50:26.900230+0000","last_became_peered":"2024-03-11T22:50:26.900230+0000","last_unstale":"2024-03-11T22:50:56.093787+0000","last_undegraded":"2024-03-11T22:50:56.093787+0000","last_fullsized":"2024-03-11T22:50:56.093787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:23:55.485115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096780+0000","last_change":"2024-03-11T22:50:28.910491+0000","last_active":"2024-03-11T22:50:56.096780+0000","last_peered":"2024-03-11T22:50:56.096780+0000","last_clean":"2024-03-11T22:50:56.096780+0000","last_became_active":"2024-03-11T22:50:28.909943+0000","last_became_peered":"2024-03-11T22:50:28.909943+0000","last_unstale":"2024-03-11T22:50:56.096780+0000","last_undegraded":"2024-03-11T22:50:56.096780+0000","last_fullsized":"2024-03-11T22:50:56.096780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:59.210238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094898+0000","last_change":"2024-03-11T22:50:30.920607+0000","last_active":"2024-03-11T22:50:56.094898+0000","last_peered":"2024-03-11T22:50:56.094898+0000","last_clean":"2024-03-11T22:50:56.094898+0000","last_became_active":"2024-03-11T22:50:30.920259+0000","last_became_peered":"2024-03-11T22:50:30.920259+0000","last_unstale":"2024-03-11T22:50:56.094898+0000","last_undegraded":"2024-03-11T22:50:56.094898+0000","last_fullsized":"2024-03-11T22:50:56.094898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:48:00.145519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36376,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704680,"statfs":{"total":95990841344,"available":95953592320,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:51:23.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:51:23.602 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:24.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:23 smithi052 ceph-mon[25824]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:24.125 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:23 smithi052 ceph-mon[25824]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:51:24.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:23 smithi116 ceph-mon[27552]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:24.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:23 smithi116 ceph-mon[27552]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:51:24.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:23 smithi152 ceph-mon[27791]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:24.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:23 smithi152 ceph-mon[27791]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:51:26.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:25 smithi052 ceph-mon[25824]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:26.215 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:51:26.217 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-03-11T22:51:26.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:25 smithi116 ceph-mon[27552]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:26.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:25 smithi152 ceph-mon[27791]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:27.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:26 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:27.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:26 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:27.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:26 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:27.564 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-11T22:51:26.109112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":100808,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":281122360,"statfs":{"total":287972524032,"available":287869296640,"internally_reserved":0,"allocated":3182592,"data_stored":2109498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002750"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095416+0000","last_change":"2024-03-11T22:50:32.923509+0000","last_active":"2024-03-11T22:50:56.095416+0000","last_peered":"2024-03-11T22:50:56.095416+0000","last_clean":"2024-03-11T22:50:56.095416+0000","last_became_active":"2024-03-11T22:50:32.923169+0000","last_became_peered":"2024-03-11T22:50:32.923169+0000","last_unstale":"2024-03-11T22:50:56.095416+0000","last_undegraded":"2024-03-11T22:50:56.095416+0000","last_fullsized":"2024-03-11T22:50:56.095416+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:07.362782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094190+0000","last_change":"2024-03-11T22:50:30.921468+0000","last_active":"2024-03-11T22:50:56.094190+0000","last_peered":"2024-03-11T22:50:56.094190+0000","last_clean":"2024-03-11T22:50:56.094190+0000","last_became_active":"2024-03-11T22:50:30.919382+0000","last_became_peered":"2024-03-11T22:50:30.919382+0000","last_unstale":"2024-03-11T22:50:56.094190+0000","last_undegraded":"2024-03-11T22:50:56.094190+0000","last_fullsized":"2024-03-11T22:50:56.094190+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:22.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095504+0000","last_change":"2024-03-11T22:50:26.899261+0000","last_active":"2024-03-11T22:50:56.095504+0000","last_peered":"2024-03-11T22:50:56.095504+0000","last_clean":"2024-03-11T22:50:56.095504+0000","last_became_active":"2024-03-11T22:50:26.898518+0000","last_became_peered":"2024-03-11T22:50:26.898518+0000","last_unstale":"2024-03-11T22:50:56.095504+0000","last_undegraded":"2024-03-11T22:50:56.095504+0000","last_fullsized":"2024-03-11T22:50:56.095504+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:34:25.851856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093659+0000","last_change":"2024-03-11T22:50:28.909151+0000","last_active":"2024-03-11T22:50:56.093659+0000","last_peered":"2024-03-11T22:50:56.093659+0000","last_clean":"2024-03-11T22:50:56.093659+0000","last_became_active":"2024-03-11T22:50:28.907933+0000","last_became_peered":"2024-03-11T22:50:28.907933+0000","last_unstale":"2024-03-11T22:50:56.093659+0000","last_undegraded":"2024-03-11T22:50:56.093659+0000","last_fullsized":"2024-03-11T22:50:56.093659+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:17:05.778171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093742+0000","last_change":"2024-03-11T22:50:32.919666+0000","last_active":"2024-03-11T22:50:56.093742+0000","last_peered":"2024-03-11T22:50:56.093742+0000","last_clean":"2024-03-11T22:50:56.093742+0000","last_became_active":"2024-03-11T22:50:32.919336+0000","last_became_peered":"2024-03-11T22:50:32.919336+0000","last_unstale":"2024-03-11T22:50:56.093742+0000","last_undegraded":"2024-03-11T22:50:56.093742+0000","last_fullsized":"2024-03-11T22:50:56.093742+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:35:55.927559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095040+0000","last_change":"2024-03-11T22:50:30.920136+0000","last_active":"2024-03-11T22:50:56.095040+0000","last_peered":"2024-03-11T22:50:56.095040+0000","last_clean":"2024-03-11T22:50:56.095040+0000","last_became_active":"2024-03-11T22:50:30.919819+0000","last_became_peered":"2024-03-11T22:50:30.919819+0000","last_unstale":"2024-03-11T22:50:56.095040+0000","last_undegraded":"2024-03-11T22:50:56.095040+0000","last_fullsized":"2024-03-11T22:50:56.095040+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:50.741318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094281+0000","last_change":"2024-03-11T22:50:26.900519+0000","last_active":"2024-03-11T22:50:56.094281+0000","last_peered":"2024-03-11T22:50:56.094281+0000","last_clean":"2024-03-11T22:50:56.094281+0000","last_became_active":"2024-03-11T22:50:26.900179+0000","last_became_peered":"2024-03-11T22:50:26.900179+0000","last_unstale":"2024-03-11T22:50:56.094281+0000","last_undegraded":"2024-03-11T22:50:56.094281+0000","last_fullsized":"2024-03-11T22:50:56.094281+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:29:32.932049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094726+0000","last_change":"2024-03-11T22:50:28.910400+0000","last_active":"2024-03-11T22:50:56.094726+0000","last_peered":"2024-03-11T22:50:56.094726+0000","last_clean":"2024-03-11T22:50:56.094726+0000","last_became_active":"2024-03-11T22:50:28.909646+0000","last_became_peered":"2024-03-11T22:50:28.909646+0000","last_unstale":"2024-03-11T22:50:56.094726+0000","last_undegraded":"2024-03-11T22:50:56.094726+0000","last_fullsized":"2024-03-11T22:50:56.094726+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:26:28.676129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094309+0000","last_change":"2024-03-11T22:50:32.916141+0000","last_active":"2024-03-11T22:50:56.094309+0000","last_peered":"2024-03-11T22:50:56.094309+0000","last_clean":"2024-03-11T22:50:56.094309+0000","last_became_active":"2024-03-11T22:50:32.915812+0000","last_became_peered":"2024-03-11T22:50:32.915812+0000","last_unstale":"2024-03-11T22:50:56.094309+0000","last_undegraded":"2024-03-11T22:50:56.094309+0000","last_fullsized":"2024-03-11T22:50:56.094309+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:30.646032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096057+0000","last_change":"2024-03-11T22:50:30.919789+0000","last_active":"2024-03-11T22:50:56.096057+0000","last_peered":"2024-03-11T22:50:56.096057+0000","last_clean":"2024-03-11T22:50:56.096057+0000","last_became_active":"2024-03-11T22:50:30.919444+0000","last_became_peered":"2024-03-11T22:50:30.919444+0000","last_unstale":"2024-03-11T22:50:56.096057+0000","last_undegraded":"2024-03-11T22:50:56.096057+0000","last_fullsized":"2024-03-11T22:50:56.096057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:58.773034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097370+0000","last_change":"2024-03-11T22:50:26.903897+0000","last_active":"2024-03-11T22:50:56.097370+0000","last_peered":"2024-03-11T22:50:56.097370+0000","last_clean":"2024-03-11T22:50:56.097370+0000","last_became_active":"2024-03-11T22:50:26.902960+0000","last_became_peered":"2024-03-11T22:50:26.902960+0000","last_unstale":"2024-03-11T22:50:56.097370+0000","last_undegraded":"2024-03-11T22:50:56.097370+0000","last_fullsized":"2024-03-11T22:50:56.097370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:30.358238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095365+0000","last_change":"2024-03-11T22:50:28.910110+0000","last_active":"2024-03-11T22:50:56.095365+0000","last_peered":"2024-03-11T22:50:56.095365+0000","last_clean":"2024-03-11T22:50:56.095365+0000","last_became_active":"2024-03-11T22:50:28.908945+0000","last_became_peered":"2024-03-11T22:50:28.908945+0000","last_unstale":"2024-03-11T22:50:56.095365+0000","last_undegraded":"2024-03-11T22:50:56.095365+0000","last_fullsized":"2024-03-11T22:50:56.095365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:51:05.096198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093805+0000","last_change":"2024-03-11T22:50:32.919705+0000","last_active":"2024-03-11T22:50:56.093805+0000","last_peered":"2024-03-11T22:50:56.093805+0000","last_clean":"2024-03-11T22:50:56.093805+0000","last_became_active":"2024-03-11T22:50:32.919260+0000","last_became_peered":"2024-03-11T22:50:32.919260+0000","last_unstale":"2024-03-11T22:50:56.093805+0000","last_undegraded":"2024-03-11T22:50:56.093805+0000","last_fullsized":"2024-03-11T22:50:56.093805+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:23:00.848480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093742+0000","last_change":"2024-03-11T22:50:30.916838+0000","last_active":"2024-03-11T22:50:56.093742+0000","last_peered":"2024-03-11T22:50:56.093742+0000","last_clean":"2024-03-11T22:50:56.093742+0000","last_became_active":"2024-03-11T22:50:30.915877+0000","last_became_peered":"2024-03-11T22:50:30.915877+0000","last_unstale":"2024-03-11T22:50:56.093742+0000","last_undegraded":"2024-03-11T22:50:56.093742+0000","last_fullsized":"2024-03-11T22:50:56.093742+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:40:20.022725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095658+0000","last_change":"2024-03-11T22:50:26.901853+0000","last_active":"2024-03-11T22:50:56.095658+0000","last_peered":"2024-03-11T22:50:56.095658+0000","last_clean":"2024-03-11T22:50:56.095658+0000","last_became_active":"2024-03-11T22:50:26.901490+0000","last_became_peered":"2024-03-11T22:50:26.901490+0000","last_unstale":"2024-03-11T22:50:56.095658+0000","last_undegraded":"2024-03-11T22:50:56.095658+0000","last_fullsized":"2024-03-11T22:50:56.095658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:41:05.428025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097241+0000","last_change":"2024-03-11T22:50:28.913248+0000","last_active":"2024-03-11T22:50:56.097241+0000","last_peered":"2024-03-11T22:50:56.097241+0000","last_clean":"2024-03-11T22:50:56.097241+0000","last_became_active":"2024-03-11T22:50:28.912856+0000","last_became_peered":"2024-03-11T22:50:28.912856+0000","last_unstale":"2024-03-11T22:50:56.097241+0000","last_undegraded":"2024-03-11T22:50:56.097241+0000","last_fullsized":"2024-03-11T22:50:56.097241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:25:29.487389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094503+0000","last_change":"2024-03-11T22:50:32.920490+0000","last_active":"2024-03-11T22:50:56.094503+0000","last_peered":"2024-03-11T22:50:56.094503+0000","last_clean":"2024-03-11T22:50:56.094503+0000","last_became_active":"2024-03-11T22:50:32.920123+0000","last_became_peered":"2024-03-11T22:50:32.920123+0000","last_unstale":"2024-03-11T22:50:56.094503+0000","last_undegraded":"2024-03-11T22:50:56.094503+0000","last_fullsized":"2024-03-11T22:50:56.094503+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:57.485435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095981+0000","last_change":"2024-03-11T22:50:30.921770+0000","last_active":"2024-03-11T22:50:56.095981+0000","last_peered":"2024-03-11T22:50:56.095981+0000","last_clean":"2024-03-11T22:50:56.095981+0000","last_became_active":"2024-03-11T22:50:30.921441+0000","last_became_peered":"2024-03-11T22:50:30.921441+0000","last_unstale":"2024-03-11T22:50:56.095981+0000","last_undegraded":"2024-03-11T22:50:56.095981+0000","last_fullsized":"2024-03-11T22:50:56.095981+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:21:45.963299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095412+0000","last_change":"2024-03-11T22:50:26.899801+0000","last_active":"2024-03-11T22:50:56.095412+0000","last_peered":"2024-03-11T22:50:56.095412+0000","last_clean":"2024-03-11T22:50:56.095412+0000","last_became_active":"2024-03-11T22:50:26.899445+0000","last_became_peered":"2024-03-11T22:50:26.899445+0000","last_unstale":"2024-03-11T22:50:56.095412+0000","last_undegraded":"2024-03-11T22:50:56.095412+0000","last_fullsized":"2024-03-11T22:50:56.095412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:01:58.268382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095484+0000","last_change":"2024-03-11T22:50:28.912720+0000","last_active":"2024-03-11T22:50:56.095484+0000","last_peered":"2024-03-11T22:50:56.095484+0000","last_clean":"2024-03-11T22:50:56.095484+0000","last_became_active":"2024-03-11T22:50:28.912422+0000","last_became_peered":"2024-03-11T22:50:28.912422+0000","last_unstale":"2024-03-11T22:50:56.095484+0000","last_undegraded":"2024-03-11T22:50:56.095484+0000","last_fullsized":"2024-03-11T22:50:56.095484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:36.835856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095617+0000","last_change":"2024-03-11T22:50:32.921678+0000","last_active":"2024-03-11T22:50:56.095617+0000","last_peered":"2024-03-11T22:50:56.095617+0000","last_clean":"2024-03-11T22:50:56.095617+0000","last_became_active":"2024-03-11T22:50:32.921260+0000","last_became_peered":"2024-03-11T22:50:32.921260+0000","last_unstale":"2024-03-11T22:50:56.095617+0000","last_undegraded":"2024-03-11T22:50:56.095617+0000","last_fullsized":"2024-03-11T22:50:56.095617+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:38:33.220835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095596+0000","last_change":"2024-03-11T22:50:30.920908+0000","last_active":"2024-03-11T22:50:56.095596+0000","last_peered":"2024-03-11T22:50:56.095596+0000","last_clean":"2024-03-11T22:50:56.095596+0000","last_became_active":"2024-03-11T22:50:30.920104+0000","last_became_peered":"2024-03-11T22:50:30.920104+0000","last_unstale":"2024-03-11T22:50:56.095596+0000","last_undegraded":"2024-03-11T22:50:56.095596+0000","last_fullsized":"2024-03-11T22:50:56.095596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:28.236707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094615+0000","last_change":"2024-03-11T22:50:26.902820+0000","last_active":"2024-03-11T22:50:56.094615+0000","last_peered":"2024-03-11T22:50:56.094615+0000","last_clean":"2024-03-11T22:50:56.094615+0000","last_became_active":"2024-03-11T22:50:26.902245+0000","last_became_peered":"2024-03-11T22:50:26.902245+0000","last_unstale":"2024-03-11T22:50:56.094615+0000","last_undegraded":"2024-03-11T22:50:56.094615+0000","last_fullsized":"2024-03-11T22:50:56.094615+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:06.302124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095137+0000","last_change":"2024-03-11T22:50:28.909065+0000","last_active":"2024-03-11T22:50:56.095137+0000","last_peered":"2024-03-11T22:50:56.095137+0000","last_clean":"2024-03-11T22:50:56.095137+0000","last_became_active":"2024-03-11T22:50:28.908728+0000","last_became_peered":"2024-03-11T22:50:28.908728+0000","last_unstale":"2024-03-11T22:50:56.095137+0000","last_undegraded":"2024-03-11T22:50:56.095137+0000","last_fullsized":"2024-03-11T22:50:56.095137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:21.757165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093244+0000","last_change":"2024-03-11T22:50:32.922608+0000","last_active":"2024-03-11T22:50:56.093244+0000","last_peered":"2024-03-11T22:50:56.093244+0000","last_clean":"2024-03-11T22:50:56.093244+0000","last_became_active":"2024-03-11T22:50:32.921834+0000","last_became_peered":"2024-03-11T22:50:32.921834+0000","last_unstale":"2024-03-11T22:50:56.093244+0000","last_undegraded":"2024-03-11T22:50:56.093244+0000","last_fullsized":"2024-03-11T22:50:56.093244+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.029611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952659+0000","last_change":"2024-03-11T22:50:30.921834+0000","last_active":"2024-03-11T22:51:16.952659+0000","last_peered":"2024-03-11T22:51:16.952659+0000","last_clean":"2024-03-11T22:51:16.952659+0000","last_became_active":"2024-03-11T22:50:30.920903+0000","last_became_peered":"2024-03-11T22:50:30.920903+0000","last_unstale":"2024-03-11T22:51:16.952659+0000","last_undegraded":"2024-03-11T22:51:16.952659+0000","last_fullsized":"2024-03-11T22:51:16.952659+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:46:42.395190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094165+0000","last_change":"2024-03-11T22:50:26.900420+0000","last_active":"2024-03-11T22:50:56.094165+0000","last_peered":"2024-03-11T22:50:56.094165+0000","last_clean":"2024-03-11T22:50:56.094165+0000","last_became_active":"2024-03-11T22:50:26.899236+0000","last_became_peered":"2024-03-11T22:50:26.899236+0000","last_unstale":"2024-03-11T22:50:56.094165+0000","last_undegraded":"2024-03-11T22:50:56.094165+0000","last_fullsized":"2024-03-11T22:50:56.094165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:02:43.949862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092528+0000","last_change":"2024-03-11T22:50:28.910125+0000","last_active":"2024-03-11T22:50:56.092528+0000","last_peered":"2024-03-11T22:50:56.092528+0000","last_clean":"2024-03-11T22:50:56.092528+0000","last_became_active":"2024-03-11T22:50:28.909772+0000","last_became_peered":"2024-03-11T22:50:28.909772+0000","last_unstale":"2024-03-11T22:50:56.092528+0000","last_undegraded":"2024-03-11T22:50:56.092528+0000","last_fullsized":"2024-03-11T22:50:56.092528+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:50.991615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096334+0000","last_change":"2024-03-11T22:50:32.923994+0000","last_active":"2024-03-11T22:50:56.096334+0000","last_peered":"2024-03-11T22:50:56.096334+0000","last_clean":"2024-03-11T22:50:56.096334+0000","last_became_active":"2024-03-11T22:50:32.923641+0000","last_became_peered":"2024-03-11T22:50:32.923641+0000","last_unstale":"2024-03-11T22:50:56.096334+0000","last_undegraded":"2024-03-11T22:50:56.096334+0000","last_fullsized":"2024-03-11T22:50:56.096334+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:22:12.491929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093405+0000","last_change":"2024-03-11T22:50:30.922334+0000","last_active":"2024-03-11T22:50:56.093405+0000","last_peered":"2024-03-11T22:50:56.093405+0000","last_clean":"2024-03-11T22:50:56.093405+0000","last_became_active":"2024-03-11T22:50:30.921916+0000","last_became_peered":"2024-03-11T22:50:30.921916+0000","last_unstale":"2024-03-11T22:50:56.093405+0000","last_undegraded":"2024-03-11T22:50:56.093405+0000","last_fullsized":"2024-03-11T22:50:56.093405+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:30.676606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094383+0000","last_change":"2024-03-11T22:50:26.901591+0000","last_active":"2024-03-11T22:50:56.094383+0000","last_peered":"2024-03-11T22:50:56.094383+0000","last_clean":"2024-03-11T22:50:56.094383+0000","last_became_active":"2024-03-11T22:50:26.900637+0000","last_became_peered":"2024-03-11T22:50:26.900637+0000","last_unstale":"2024-03-11T22:50:56.094383+0000","last_undegraded":"2024-03-11T22:50:56.094383+0000","last_fullsized":"2024-03-11T22:50:56.094383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:25:27.033569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092765+0000","last_change":"2024-03-11T22:50:28.908682+0000","last_active":"2024-03-11T22:50:56.092765+0000","last_peered":"2024-03-11T22:50:56.092765+0000","last_clean":"2024-03-11T22:50:56.092765+0000","last_became_active":"2024-03-11T22:50:28.908408+0000","last_became_peered":"2024-03-11T22:50:28.908408+0000","last_unstale":"2024-03-11T22:50:56.092765+0000","last_undegraded":"2024-03-11T22:50:56.092765+0000","last_fullsized":"2024-03-11T22:50:56.092765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:19.041424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095257+0000","last_change":"2024-03-11T22:50:32.917075+0000","last_active":"2024-03-11T22:50:56.095257+0000","last_peered":"2024-03-11T22:50:56.095257+0000","last_clean":"2024-03-11T22:50:56.095257+0000","last_became_active":"2024-03-11T22:50:32.916756+0000","last_became_peered":"2024-03-11T22:50:32.916756+0000","last_unstale":"2024-03-11T22:50:56.095257+0000","last_undegraded":"2024-03-11T22:50:56.095257+0000","last_fullsized":"2024-03-11T22:50:56.095257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:47.892516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092588+0000","last_change":"2024-03-11T22:50:30.912428+0000","last_active":"2024-03-11T22:50:56.092588+0000","last_peered":"2024-03-11T22:50:56.092588+0000","last_clean":"2024-03-11T22:50:56.092588+0000","last_became_active":"2024-03-11T22:50:30.912052+0000","last_became_peered":"2024-03-11T22:50:30.912052+0000","last_unstale":"2024-03-11T22:50:56.092588+0000","last_undegraded":"2024-03-11T22:50:56.092588+0000","last_fullsized":"2024-03-11T22:50:56.092588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:22:10.011551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096973+0000","last_change":"2024-03-11T22:50:26.902823+0000","last_active":"2024-03-11T22:50:56.096973+0000","last_peered":"2024-03-11T22:50:56.096973+0000","last_clean":"2024-03-11T22:50:56.096973+0000","last_became_active":"2024-03-11T22:50:26.902410+0000","last_became_peered":"2024-03-11T22:50:26.902410+0000","last_unstale":"2024-03-11T22:50:56.096973+0000","last_undegraded":"2024-03-11T22:50:56.096973+0000","last_fullsized":"2024-03-11T22:50:56.096973+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:55:27.791788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096368+0000","last_change":"2024-03-11T22:50:28.911574+0000","last_active":"2024-03-11T22:50:56.096368+0000","last_peered":"2024-03-11T22:50:56.096368+0000","last_clean":"2024-03-11T22:50:56.096368+0000","last_became_active":"2024-03-11T22:50:28.910483+0000","last_became_peered":"2024-03-11T22:50:28.910483+0000","last_unstale":"2024-03-11T22:50:56.096368+0000","last_undegraded":"2024-03-11T22:50:56.096368+0000","last_fullsized":"2024-03-11T22:50:56.096368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:01.197107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093784+0000","last_change":"2024-03-11T22:50:32.917035+0000","last_active":"2024-03-11T22:50:56.093784+0000","last_peered":"2024-03-11T22:50:56.093784+0000","last_clean":"2024-03-11T22:50:56.093784+0000","last_became_active":"2024-03-11T22:50:32.916689+0000","last_became_peered":"2024-03-11T22:50:32.916689+0000","last_unstale":"2024-03-11T22:50:56.093784+0000","last_undegraded":"2024-03-11T22:50:56.093784+0000","last_fullsized":"2024-03-11T22:50:56.093784+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:07:25.780960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092914+0000","last_change":"2024-03-11T22:50:30.921519+0000","last_active":"2024-03-11T22:50:56.092914+0000","last_peered":"2024-03-11T22:50:56.092914+0000","last_clean":"2024-03-11T22:50:56.092914+0000","last_became_active":"2024-03-11T22:50:30.921083+0000","last_became_peered":"2024-03-11T22:50:30.921083+0000","last_unstale":"2024-03-11T22:50:56.092914+0000","last_undegraded":"2024-03-11T22:50:56.092914+0000","last_fullsized":"2024-03-11T22:50:56.092914+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:49:52.461231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096524+0000","last_change":"2024-03-11T22:50:26.904050+0000","last_active":"2024-03-11T22:50:56.096524+0000","last_peered":"2024-03-11T22:50:56.096524+0000","last_clean":"2024-03-11T22:50:56.096524+0000","last_became_active":"2024-03-11T22:50:26.903615+0000","last_became_peered":"2024-03-11T22:50:26.903615+0000","last_unstale":"2024-03-11T22:50:56.096524+0000","last_undegraded":"2024-03-11T22:50:56.096524+0000","last_fullsized":"2024-03-11T22:50:56.096524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:31:07.661888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095631+0000","last_change":"2024-03-11T22:50:28.910648+0000","last_active":"2024-03-11T22:50:56.095631+0000","last_peered":"2024-03-11T22:50:56.095631+0000","last_clean":"2024-03-11T22:50:56.095631+0000","last_became_active":"2024-03-11T22:50:28.910032+0000","last_became_peered":"2024-03-11T22:50:28.910032+0000","last_unstale":"2024-03-11T22:50:56.095631+0000","last_undegraded":"2024-03-11T22:50:56.095631+0000","last_fullsized":"2024-03-11T22:50:56.095631+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:17:36.534219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094500+0000","last_change":"2024-03-11T22:50:32.919497+0000","last_active":"2024-03-11T22:50:56.094500+0000","last_peered":"2024-03-11T22:50:56.094500+0000","last_clean":"2024-03-11T22:50:56.094500+0000","last_became_active":"2024-03-11T22:50:32.918980+0000","last_became_peered":"2024-03-11T22:50:32.918980+0000","last_unstale":"2024-03-11T22:50:56.094500+0000","last_undegraded":"2024-03-11T22:50:56.094500+0000","last_fullsized":"2024-03-11T22:50:56.094500+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:57:23.223957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096217+0000","last_change":"2024-03-11T22:50:30.919841+0000","last_active":"2024-03-11T22:50:56.096217+0000","last_peered":"2024-03-11T22:50:56.096217+0000","last_clean":"2024-03-11T22:50:56.096217+0000","last_became_active":"2024-03-11T22:50:30.919160+0000","last_became_peered":"2024-03-11T22:50:30.919160+0000","last_unstale":"2024-03-11T22:50:56.096217+0000","last_undegraded":"2024-03-11T22:50:56.096217+0000","last_fullsized":"2024-03-11T22:50:56.096217+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:33.341312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097185+0000","last_change":"2024-03-11T22:50:26.903671+0000","last_active":"2024-03-11T22:50:56.097185+0000","last_peered":"2024-03-11T22:50:56.097185+0000","last_clean":"2024-03-11T22:50:56.097185+0000","last_became_active":"2024-03-11T22:50:26.901916+0000","last_became_peered":"2024-03-11T22:50:26.901916+0000","last_unstale":"2024-03-11T22:50:56.097185+0000","last_undegraded":"2024-03-11T22:50:56.097185+0000","last_fullsized":"2024-03-11T22:50:56.097185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:46:11.568818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095069+0000","last_change":"2024-03-11T22:50:28.911686+0000","last_active":"2024-03-11T22:50:56.095069+0000","last_peered":"2024-03-11T22:50:56.095069+0000","last_clean":"2024-03-11T22:50:56.095069+0000","last_became_active":"2024-03-11T22:50:28.910611+0000","last_became_peered":"2024-03-11T22:50:28.910611+0000","last_unstale":"2024-03-11T22:50:56.095069+0000","last_undegraded":"2024-03-11T22:50:56.095069+0000","last_fullsized":"2024-03-11T22:50:56.095069+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:27:55.181948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094464+0000","last_change":"2024-03-11T22:50:32.919512+0000","last_active":"2024-03-11T22:50:56.094464+0000","last_peered":"2024-03-11T22:50:56.094464+0000","last_clean":"2024-03-11T22:50:56.094464+0000","last_became_active":"2024-03-11T22:50:32.918973+0000","last_became_peered":"2024-03-11T22:50:32.918973+0000","last_unstale":"2024-03-11T22:50:56.094464+0000","last_undegraded":"2024-03-11T22:50:56.094464+0000","last_fullsized":"2024-03-11T22:50:56.094464+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:25.237547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092861+0000","last_change":"2024-03-11T22:50:30.914227+0000","last_active":"2024-03-11T22:50:56.092861+0000","last_peered":"2024-03-11T22:50:56.092861+0000","last_clean":"2024-03-11T22:50:56.092861+0000","last_became_active":"2024-03-11T22:50:30.913851+0000","last_became_peered":"2024-03-11T22:50:30.913851+0000","last_unstale":"2024-03-11T22:50:56.092861+0000","last_undegraded":"2024-03-11T22:50:56.092861+0000","last_fullsized":"2024-03-11T22:50:56.092861+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:17.633666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094843+0000","last_change":"2024-03-11T22:50:26.903625+0000","last_active":"2024-03-11T22:50:56.094843+0000","last_peered":"2024-03-11T22:50:56.094843+0000","last_clean":"2024-03-11T22:50:56.094843+0000","last_became_active":"2024-03-11T22:50:26.903206+0000","last_became_peered":"2024-03-11T22:50:26.903206+0000","last_unstale":"2024-03-11T22:50:56.094843+0000","last_undegraded":"2024-03-11T22:50:56.094843+0000","last_fullsized":"2024-03-11T22:50:56.094843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:03:27.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096413+0000","last_change":"2024-03-11T22:50:28.913328+0000","last_active":"2024-03-11T22:50:56.096413+0000","last_peered":"2024-03-11T22:50:56.096413+0000","last_clean":"2024-03-11T22:50:56.096413+0000","last_became_active":"2024-03-11T22:50:28.912923+0000","last_became_peered":"2024-03-11T22:50:28.912923+0000","last_unstale":"2024-03-11T22:50:56.096413+0000","last_undegraded":"2024-03-11T22:50:56.096413+0000","last_fullsized":"2024-03-11T22:50:56.096413+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:32:54.273252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093941+0000","last_change":"2024-03-11T22:50:32.921655+0000","last_active":"2024-03-11T22:50:56.093941+0000","last_peered":"2024-03-11T22:50:56.093941+0000","last_clean":"2024-03-11T22:50:56.093941+0000","last_became_active":"2024-03-11T22:50:32.920982+0000","last_became_peered":"2024-03-11T22:50:32.920982+0000","last_unstale":"2024-03-11T22:50:56.093941+0000","last_undegraded":"2024-03-11T22:50:56.093941+0000","last_fullsized":"2024-03-11T22:50:56.093941+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:31:45.726492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952689+0000","last_change":"2024-03-11T22:50:30.921133+0000","last_active":"2024-03-11T22:51:16.952689+0000","last_peered":"2024-03-11T22:51:16.952689+0000","last_clean":"2024-03-11T22:51:16.952689+0000","last_became_active":"2024-03-11T22:50:30.920504+0000","last_became_peered":"2024-03-11T22:50:30.920504+0000","last_unstale":"2024-03-11T22:51:16.952689+0000","last_undegraded":"2024-03-11T22:51:16.952689+0000","last_fullsized":"2024-03-11T22:51:16.952689+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:00:31.672875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095045+0000","last_change":"2024-03-11T22:50:26.901345+0000","last_active":"2024-03-11T22:50:56.095045+0000","last_peered":"2024-03-11T22:50:56.095045+0000","last_clean":"2024-03-11T22:50:56.095045+0000","last_became_active":"2024-03-11T22:50:26.900653+0000","last_became_peered":"2024-03-11T22:50:26.900653+0000","last_unstale":"2024-03-11T22:50:56.095045+0000","last_undegraded":"2024-03-11T22:50:56.095045+0000","last_fullsized":"2024-03-11T22:50:56.095045+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:17:48.520316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093392+0000","last_change":"2024-03-11T22:50:28.910131+0000","last_active":"2024-03-11T22:50:56.093392+0000","last_peered":"2024-03-11T22:50:56.093392+0000","last_clean":"2024-03-11T22:50:56.093392+0000","last_became_active":"2024-03-11T22:50:28.909751+0000","last_became_peered":"2024-03-11T22:50:28.909751+0000","last_unstale":"2024-03-11T22:50:56.093392+0000","last_undegraded":"2024-03-11T22:50:56.093392+0000","last_fullsized":"2024-03-11T22:50:56.093392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:21.451987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096157+0000","last_change":"2024-03-11T22:50:32.920843+0000","last_active":"2024-03-11T22:50:56.096157+0000","last_peered":"2024-03-11T22:50:56.096157+0000","last_clean":"2024-03-11T22:50:56.096157+0000","last_became_active":"2024-03-11T22:50:32.920329+0000","last_became_peered":"2024-03-11T22:50:32.920329+0000","last_unstale":"2024-03-11T22:50:56.096157+0000","last_undegraded":"2024-03-11T22:50:56.096157+0000","last_fullsized":"2024-03-11T22:50:56.096157+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:19:40.858880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952730+0000","last_change":"2024-03-11T22:50:30.920943+0000","last_active":"2024-03-11T22:51:16.952730+0000","last_peered":"2024-03-11T22:51:16.952730+0000","last_clean":"2024-03-11T22:51:16.952730+0000","last_became_active":"2024-03-11T22:50:30.920647+0000","last_became_peered":"2024-03-11T22:50:30.920647+0000","last_unstale":"2024-03-11T22:51:16.952730+0000","last_undegraded":"2024-03-11T22:51:16.952730+0000","last_fullsized":"2024-03-11T22:51:16.952730+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:00:48.958373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094762+0000","last_change":"2024-03-11T22:50:26.901176+0000","last_active":"2024-03-11T22:50:56.094762+0000","last_peered":"2024-03-11T22:50:56.094762+0000","last_clean":"2024-03-11T22:50:56.094762+0000","last_became_active":"2024-03-11T22:50:26.900572+0000","last_became_peered":"2024-03-11T22:50:26.900572+0000","last_unstale":"2024-03-11T22:50:56.094762+0000","last_undegraded":"2024-03-11T22:50:56.094762+0000","last_fullsized":"2024-03-11T22:50:56.094762+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:48:18.275141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096443+0000","last_change":"2024-03-11T22:50:28.910277+0000","last_active":"2024-03-11T22:50:56.096443+0000","last_peered":"2024-03-11T22:50:56.096443+0000","last_clean":"2024-03-11T22:50:56.096443+0000","last_became_active":"2024-03-11T22:50:28.909734+0000","last_became_peered":"2024-03-11T22:50:28.909734+0000","last_unstale":"2024-03-11T22:50:56.096443+0000","last_undegraded":"2024-03-11T22:50:56.096443+0000","last_fullsized":"2024-03-11T22:50:56.096443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:45.990785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094376+0000","last_change":"2024-03-11T22:50:32.924198+0000","last_active":"2024-03-11T22:50:56.094376+0000","last_peered":"2024-03-11T22:50:56.094376+0000","last_clean":"2024-03-11T22:50:56.094376+0000","last_became_active":"2024-03-11T22:50:32.923802+0000","last_became_peered":"2024-03-11T22:50:32.923802+0000","last_unstale":"2024-03-11T22:50:56.094376+0000","last_undegraded":"2024-03-11T22:50:56.094376+0000","last_fullsized":"2024-03-11T22:50:56.094376+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:46.861200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094564+0000","last_change":"2024-03-11T22:50:30.921967+0000","last_active":"2024-03-11T22:50:56.094564+0000","last_peered":"2024-03-11T22:50:56.094564+0000","last_clean":"2024-03-11T22:50:56.094564+0000","last_became_active":"2024-03-11T22:50:30.921589+0000","last_became_peered":"2024-03-11T22:50:30.921589+0000","last_unstale":"2024-03-11T22:50:56.094564+0000","last_undegraded":"2024-03-11T22:50:56.094564+0000","last_fullsized":"2024-03-11T22:50:56.094564+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:57:44.858552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093909+0000","last_change":"2024-03-11T22:50:26.904482+0000","last_active":"2024-03-11T22:50:56.093909+0000","last_peered":"2024-03-11T22:50:56.093909+0000","last_clean":"2024-03-11T22:50:56.093909+0000","last_became_active":"2024-03-11T22:50:26.904032+0000","last_became_peered":"2024-03-11T22:50:26.904032+0000","last_unstale":"2024-03-11T22:50:56.093909+0000","last_undegraded":"2024-03-11T22:50:56.093909+0000","last_fullsized":"2024-03-11T22:50:56.093909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:32:39.080934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093623+0000","last_change":"2024-03-11T22:50:28.911457+0000","last_active":"2024-03-11T22:50:56.093623+0000","last_peered":"2024-03-11T22:50:56.093623+0000","last_clean":"2024-03-11T22:50:56.093623+0000","last_became_active":"2024-03-11T22:50:28.909930+0000","last_became_peered":"2024-03-11T22:50:28.909930+0000","last_unstale":"2024-03-11T22:50:56.093623+0000","last_undegraded":"2024-03-11T22:50:56.093623+0000","last_fullsized":"2024-03-11T22:50:56.093623+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:26.804925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094245+0000","last_change":"2024-03-11T22:50:32.920993+0000","last_active":"2024-03-11T22:50:56.094245+0000","last_peered":"2024-03-11T22:50:56.094245+0000","last_clean":"2024-03-11T22:50:56.094245+0000","last_became_active":"2024-03-11T22:50:32.920546+0000","last_became_peered":"2024-03-11T22:50:32.920546+0000","last_unstale":"2024-03-11T22:50:56.094245+0000","last_undegraded":"2024-03-11T22:50:56.094245+0000","last_fullsized":"2024-03-11T22:50:56.094245+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:36:37.571372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093498+0000","last_change":"2024-03-11T22:50:30.921565+0000","last_active":"2024-03-11T22:50:56.093498+0000","last_peered":"2024-03-11T22:50:56.093498+0000","last_clean":"2024-03-11T22:50:56.093498+0000","last_became_active":"2024-03-11T22:50:30.920617+0000","last_became_peered":"2024-03-11T22:50:30.920617+0000","last_unstale":"2024-03-11T22:50:56.093498+0000","last_undegraded":"2024-03-11T22:50:56.093498+0000","last_fullsized":"2024-03-11T22:50:56.093498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:39.490798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094528+0000","last_change":"2024-03-11T22:50:26.901643+0000","last_active":"2024-03-11T22:50:56.094528+0000","last_peered":"2024-03-11T22:50:56.094528+0000","last_clean":"2024-03-11T22:50:56.094528+0000","last_became_active":"2024-03-11T22:50:26.901029+0000","last_became_peered":"2024-03-11T22:50:26.901029+0000","last_unstale":"2024-03-11T22:50:56.094528+0000","last_undegraded":"2024-03-11T22:50:56.094528+0000","last_fullsized":"2024-03-11T22:50:56.094528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:10:11.840759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095375+0000","last_change":"2024-03-11T22:50:28.912593+0000","last_active":"2024-03-11T22:50:56.095375+0000","last_peered":"2024-03-11T22:50:56.095375+0000","last_clean":"2024-03-11T22:50:56.095375+0000","last_became_active":"2024-03-11T22:50:28.912297+0000","last_became_peered":"2024-03-11T22:50:28.912297+0000","last_unstale":"2024-03-11T22:50:56.095375+0000","last_undegraded":"2024-03-11T22:50:56.095375+0000","last_fullsized":"2024-03-11T22:50:56.095375+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:23.501238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095360+0000","last_change":"2024-03-11T22:50:32.917771+0000","last_active":"2024-03-11T22:50:56.095360+0000","last_peered":"2024-03-11T22:50:56.095360+0000","last_clean":"2024-03-11T22:50:56.095360+0000","last_became_active":"2024-03-11T22:50:32.917545+0000","last_became_peered":"2024-03-11T22:50:32.917545+0000","last_unstale":"2024-03-11T22:50:56.095360+0000","last_undegraded":"2024-03-11T22:50:56.095360+0000","last_fullsized":"2024-03-11T22:50:56.095360+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:34.091143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097195+0000","last_change":"2024-03-11T22:50:26.903500+0000","last_active":"2024-03-11T22:50:56.097195+0000","last_peered":"2024-03-11T22:50:56.097195+0000","last_clean":"2024-03-11T22:50:56.097195+0000","last_became_active":"2024-03-11T22:50:26.902331+0000","last_became_peered":"2024-03-11T22:50:26.902331+0000","last_unstale":"2024-03-11T22:50:56.097195+0000","last_undegraded":"2024-03-11T22:50:56.097195+0000","last_fullsized":"2024-03-11T22:50:56.097195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:40.049766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.097060+0000","last_change":"2024-03-11T22:50:28.910021+0000","last_active":"2024-03-11T22:50:56.097060+0000","last_peered":"2024-03-11T22:50:56.097060+0000","last_clean":"2024-03-11T22:50:56.097060+0000","last_became_active":"2024-03-11T22:50:28.909735+0000","last_became_peered":"2024-03-11T22:50:28.909735+0000","last_unstale":"2024-03-11T22:50:56.097060+0000","last_undegraded":"2024-03-11T22:50:56.097060+0000","last_fullsized":"2024-03-11T22:50:56.097060+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:35:50.874827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952740+0000","last_change":"2024-03-11T22:50:30.919739+0000","last_active":"2024-03-11T22:51:16.952740+0000","last_peered":"2024-03-11T22:51:16.952740+0000","last_clean":"2024-03-11T22:51:16.952740+0000","last_became_active":"2024-03-11T22:50:30.919276+0000","last_became_peered":"2024-03-11T22:50:30.919276+0000","last_unstale":"2024-03-11T22:51:16.952740+0000","last_undegraded":"2024-03-11T22:51:16.952740+0000","last_fullsized":"2024-03-11T22:51:16.952740+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:22:54.554795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093024+0000","last_change":"2024-03-11T22:50:32.918992+0000","last_active":"2024-03-11T22:50:56.093024+0000","last_peered":"2024-03-11T22:50:56.093024+0000","last_clean":"2024-03-11T22:50:56.093024+0000","last_became_active":"2024-03-11T22:50:32.917545+0000","last_became_peered":"2024-03-11T22:50:32.917545+0000","last_unstale":"2024-03-11T22:50:56.093024+0000","last_undegraded":"2024-03-11T22:50:56.093024+0000","last_fullsized":"2024-03-11T22:50:56.093024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:12:58.613786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094164+0000","last_change":"2024-03-11T22:50:26.899067+0000","last_active":"2024-03-11T22:50:56.094164+0000","last_peered":"2024-03-11T22:50:56.094164+0000","last_clean":"2024-03-11T22:50:56.094164+0000","last_became_active":"2024-03-11T22:50:26.898249+0000","last_became_peered":"2024-03-11T22:50:26.898249+0000","last_unstale":"2024-03-11T22:50:56.094164+0000","last_undegraded":"2024-03-11T22:50:56.094164+0000","last_fullsized":"2024-03-11T22:50:56.094164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:27.893635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":57,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952695+0000","last_change":"2024-03-11T22:50:28.908954+0000","last_active":"2024-03-11T22:51:16.952695+0000","last_peered":"2024-03-11T22:51:16.952695+0000","last_clean":"2024-03-11T22:51:16.952695+0000","last_became_active":"2024-03-11T22:50:28.908195+0000","last_became_peered":"2024-03-11T22:50:28.908195+0000","last_unstale":"2024-03-11T22:51:16.952695+0000","last_undegraded":"2024-03-11T22:51:16.952695+0000","last_fullsized":"2024-03-11T22:51:16.952695+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:47:48.956413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094599+0000","last_change":"2024-03-11T22:50:30.920139+0000","last_active":"2024-03-11T22:50:56.094599+0000","last_peered":"2024-03-11T22:50:56.094599+0000","last_clean":"2024-03-11T22:50:56.094599+0000","last_became_active":"2024-03-11T22:50:30.919701+0000","last_became_peered":"2024-03-11T22:50:30.919701+0000","last_unstale":"2024-03-11T22:50:56.094599+0000","last_undegraded":"2024-03-11T22:50:56.094599+0000","last_fullsized":"2024-03-11T22:50:56.094599+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:18:39.520668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094385+0000","last_change":"2024-03-11T22:50:32.921282+0000","last_active":"2024-03-11T22:50:56.094385+0000","last_peered":"2024-03-11T22:50:56.094385+0000","last_clean":"2024-03-11T22:50:56.094385+0000","last_became_active":"2024-03-11T22:50:32.920902+0000","last_became_peered":"2024-03-11T22:50:32.920902+0000","last_unstale":"2024-03-11T22:50:56.094385+0000","last_undegraded":"2024-03-11T22:50:56.094385+0000","last_fullsized":"2024-03-11T22:50:56.094385+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:11:32.247011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094761+0000","last_change":"2024-03-11T22:50:26.902811+0000","last_active":"2024-03-11T22:50:56.094761+0000","last_peered":"2024-03-11T22:50:56.094761+0000","last_clean":"2024-03-11T22:50:56.094761+0000","last_became_active":"2024-03-11T22:50:26.902371+0000","last_became_peered":"2024-03-11T22:50:26.902371+0000","last_unstale":"2024-03-11T22:50:56.094761+0000","last_undegraded":"2024-03-11T22:50:56.094761+0000","last_fullsized":"2024-03-11T22:50:56.094761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:03:23.892861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096848+0000","last_change":"2024-03-11T22:50:28.909098+0000","last_active":"2024-03-11T22:50:56.096848+0000","last_peered":"2024-03-11T22:50:56.096848+0000","last_clean":"2024-03-11T22:50:56.096848+0000","last_became_active":"2024-03-11T22:50:28.908451+0000","last_became_peered":"2024-03-11T22:50:28.908451+0000","last_unstale":"2024-03-11T22:50:56.096848+0000","last_undegraded":"2024-03-11T22:50:56.096848+0000","last_fullsized":"2024-03-11T22:50:56.096848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:47.483886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094043+0000","last_change":"2024-03-11T22:50:30.920511+0000","last_active":"2024-03-11T22:50:56.094043+0000","last_peered":"2024-03-11T22:50:56.094043+0000","last_clean":"2024-03-11T22:50:56.094043+0000","last_became_active":"2024-03-11T22:50:30.920186+0000","last_became_peered":"2024-03-11T22:50:30.920186+0000","last_unstale":"2024-03-11T22:50:56.094043+0000","last_undegraded":"2024-03-11T22:50:56.094043+0000","last_fullsized":"2024-03-11T22:50:56.094043+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:47.625083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094375+0000","last_change":"2024-03-11T22:50:32.924799+0000","last_active":"2024-03-11T22:50:56.094375+0000","last_peered":"2024-03-11T22:50:56.094375+0000","last_clean":"2024-03-11T22:50:56.094375+0000","last_became_active":"2024-03-11T22:50:32.924511+0000","last_became_peered":"2024-03-11T22:50:32.924511+0000","last_unstale":"2024-03-11T22:50:56.094375+0000","last_undegraded":"2024-03-11T22:50:56.094375+0000","last_fullsized":"2024-03-11T22:50:56.094375+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:55.781975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094145+0000","last_change":"2024-03-11T22:50:28.912915+0000","last_active":"2024-03-11T22:50:56.094145+0000","last_peered":"2024-03-11T22:50:56.094145+0000","last_clean":"2024-03-11T22:50:56.094145+0000","last_became_active":"2024-03-11T22:50:28.912648+0000","last_became_peered":"2024-03-11T22:50:28.912648+0000","last_unstale":"2024-03-11T22:50:56.094145+0000","last_undegraded":"2024-03-11T22:50:56.094145+0000","last_fullsized":"2024-03-11T22:50:56.094145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:11:22.591244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094010+0000","last_change":"2024-03-11T22:50:26.900122+0000","last_active":"2024-03-11T22:50:56.094010+0000","last_peered":"2024-03-11T22:50:56.094010+0000","last_clean":"2024-03-11T22:50:56.094010+0000","last_became_active":"2024-03-11T22:50:26.899230+0000","last_became_peered":"2024-03-11T22:50:26.899230+0000","last_unstale":"2024-03-11T22:50:56.094010+0000","last_undegraded":"2024-03-11T22:50:56.094010+0000","last_fullsized":"2024-03-11T22:50:56.094010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:27:04.428241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095594+0000","last_change":"2024-03-11T22:50:30.920899+0000","last_active":"2024-03-11T22:50:56.095594+0000","last_peered":"2024-03-11T22:50:56.095594+0000","last_clean":"2024-03-11T22:50:56.095594+0000","last_became_active":"2024-03-11T22:50:30.920259+0000","last_became_peered":"2024-03-11T22:50:30.920259+0000","last_unstale":"2024-03-11T22:50:56.095594+0000","last_undegraded":"2024-03-11T22:50:56.095594+0000","last_fullsized":"2024-03-11T22:50:56.095594+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:32:38.776158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095267+0000","last_change":"2024-03-11T22:50:32.919764+0000","last_active":"2024-03-11T22:50:56.095267+0000","last_peered":"2024-03-11T22:50:56.095267+0000","last_clean":"2024-03-11T22:50:56.095267+0000","last_became_active":"2024-03-11T22:50:32.919442+0000","last_became_peered":"2024-03-11T22:50:32.919442+0000","last_unstale":"2024-03-11T22:50:56.095267+0000","last_undegraded":"2024-03-11T22:50:56.095267+0000","last_fullsized":"2024-03-11T22:50:56.095267+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:52:02.533788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093141+0000","last_change":"2024-03-11T22:50:28.908575+0000","last_active":"2024-03-11T22:50:56.093141+0000","last_peered":"2024-03-11T22:50:56.093141+0000","last_clean":"2024-03-11T22:50:56.093141+0000","last_became_active":"2024-03-11T22:50:28.908093+0000","last_became_peered":"2024-03-11T22:50:28.908093+0000","last_unstale":"2024-03-11T22:50:56.093141+0000","last_undegraded":"2024-03-11T22:50:56.093141+0000","last_fullsized":"2024-03-11T22:50:56.093141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:54:13.708444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096581+0000","last_change":"2024-03-11T22:50:26.902961+0000","last_active":"2024-03-11T22:50:56.096581+0000","last_peered":"2024-03-11T22:50:56.096581+0000","last_clean":"2024-03-11T22:50:56.096581+0000","last_became_active":"2024-03-11T22:50:26.902420+0000","last_became_peered":"2024-03-11T22:50:26.902420+0000","last_unstale":"2024-03-11T22:50:56.096581+0000","last_undegraded":"2024-03-11T22:50:56.096581+0000","last_fullsized":"2024-03-11T22:50:56.096581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:47:19.560282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:58.203664+0000","last_change":"2024-03-11T22:50:15.659487+0000","last_active":"2024-03-11T22:50:58.203664+0000","last_peered":"2024-03-11T22:50:58.203664+0000","last_clean":"2024-03-11T22:50:58.203664+0000","last_became_active":"2024-03-11T22:50:15.659173+0000","last_became_peered":"2024-03-11T22:50:15.659173+0000","last_unstale":"2024-03-11T22:50:58.203664+0000","last_undegraded":"2024-03-11T22:50:58.203664+0000","last_fullsized":"2024-03-11T22:50:58.203664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:14.645525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:14.645525+0000","last_clean_scrub_stamp":"2024-03-11T22:50:14.645525+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:30.563534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094642+0000","last_change":"2024-03-11T22:50:30.916891+0000","last_active":"2024-03-11T22:50:56.094642+0000","last_peered":"2024-03-11T22:50:56.094642+0000","last_clean":"2024-03-11T22:50:56.094642+0000","last_became_active":"2024-03-11T22:50:30.915600+0000","last_became_peered":"2024-03-11T22:50:30.915600+0000","last_unstale":"2024-03-11T22:50:56.094642+0000","last_undegraded":"2024-03-11T22:50:56.094642+0000","last_fullsized":"2024-03-11T22:50:56.094642+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:21:45.576873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096512+0000","last_change":"2024-03-11T22:50:32.920993+0000","last_active":"2024-03-11T22:50:56.096512+0000","last_peered":"2024-03-11T22:50:56.096512+0000","last_clean":"2024-03-11T22:50:56.096512+0000","last_became_active":"2024-03-11T22:50:32.920596+0000","last_became_peered":"2024-03-11T22:50:32.920596+0000","last_unstale":"2024-03-11T22:50:56.096512+0000","last_undegraded":"2024-03-11T22:50:56.096512+0000","last_fullsized":"2024-03-11T22:50:56.096512+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:32:41.364871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095838+0000","last_change":"2024-03-11T22:50:28.908640+0000","last_active":"2024-03-11T22:50:56.095838+0000","last_peered":"2024-03-11T22:50:56.095838+0000","last_clean":"2024-03-11T22:50:56.095838+0000","last_became_active":"2024-03-11T22:50:28.908295+0000","last_became_peered":"2024-03-11T22:50:28.908295+0000","last_unstale":"2024-03-11T22:50:56.095838+0000","last_undegraded":"2024-03-11T22:50:56.095838+0000","last_fullsized":"2024-03-11T22:50:56.095838+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:08.294780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093576+0000","last_change":"2024-03-11T22:50:26.904372+0000","last_active":"2024-03-11T22:50:56.093576+0000","last_peered":"2024-03-11T22:50:56.093576+0000","last_clean":"2024-03-11T22:50:56.093576+0000","last_became_active":"2024-03-11T22:50:26.903883+0000","last_became_peered":"2024-03-11T22:50:26.903883+0000","last_unstale":"2024-03-11T22:50:56.093576+0000","last_undegraded":"2024-03-11T22:50:56.093576+0000","last_fullsized":"2024-03-11T22:50:56.093576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:36:59.664071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096255+0000","last_change":"2024-03-11T22:50:30.919794+0000","last_active":"2024-03-11T22:50:56.096255+0000","last_peered":"2024-03-11T22:50:56.096255+0000","last_clean":"2024-03-11T22:50:56.096255+0000","last_became_active":"2024-03-11T22:50:30.919455+0000","last_became_peered":"2024-03-11T22:50:30.919455+0000","last_unstale":"2024-03-11T22:50:56.096255+0000","last_undegraded":"2024-03-11T22:50:56.096255+0000","last_fullsized":"2024-03-11T22:50:56.096255+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:33.535853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094531+0000","last_change":"2024-03-11T22:50:32.919924+0000","last_active":"2024-03-11T22:50:56.094531+0000","last_peered":"2024-03-11T22:50:56.094531+0000","last_clean":"2024-03-11T22:50:56.094531+0000","last_became_active":"2024-03-11T22:50:32.919494+0000","last_became_peered":"2024-03-11T22:50:32.919494+0000","last_unstale":"2024-03-11T22:50:56.094531+0000","last_undegraded":"2024-03-11T22:50:56.094531+0000","last_fullsized":"2024-03-11T22:50:56.094531+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:42:14.537886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096465+0000","last_change":"2024-03-11T22:50:26.903712+0000","last_active":"2024-03-11T22:50:56.096465+0000","last_peered":"2024-03-11T22:50:56.096465+0000","last_clean":"2024-03-11T22:50:56.096465+0000","last_became_active":"2024-03-11T22:50:26.902432+0000","last_became_peered":"2024-03-11T22:50:26.902432+0000","last_unstale":"2024-03-11T22:50:56.096465+0000","last_undegraded":"2024-03-11T22:50:56.096465+0000","last_fullsized":"2024-03-11T22:50:56.096465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:06.577403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":83,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952380+0000","last_change":"2024-03-11T22:50:28.909230+0000","last_active":"2024-03-11T22:51:16.952380+0000","last_peered":"2024-03-11T22:51:16.952380+0000","last_clean":"2024-03-11T22:51:16.952380+0000","last_became_active":"2024-03-11T22:50:28.908935+0000","last_became_peered":"2024-03-11T22:50:28.908935+0000","last_unstale":"2024-03-11T22:51:16.952380+0000","last_undegraded":"2024-03-11T22:51:16.952380+0000","last_fullsized":"2024-03-11T22:51:16.952380+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:47:44.057404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952669+0000","last_change":"2024-03-11T22:50:30.921404+0000","last_active":"2024-03-11T22:51:16.952669+0000","last_peered":"2024-03-11T22:51:16.952669+0000","last_clean":"2024-03-11T22:51:16.952669+0000","last_became_active":"2024-03-11T22:50:30.921084+0000","last_became_peered":"2024-03-11T22:50:30.921084+0000","last_unstale":"2024-03-11T22:51:16.952669+0000","last_undegraded":"2024-03-11T22:51:16.952669+0000","last_fullsized":"2024-03-11T22:51:16.952669+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:58:25.723151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093533+0000","last_change":"2024-03-11T22:50:32.920690+0000","last_active":"2024-03-11T22:50:56.093533+0000","last_peered":"2024-03-11T22:50:56.093533+0000","last_clean":"2024-03-11T22:50:56.093533+0000","last_became_active":"2024-03-11T22:50:32.919747+0000","last_became_peered":"2024-03-11T22:50:32.919747+0000","last_unstale":"2024-03-11T22:50:56.093533+0000","last_undegraded":"2024-03-11T22:50:56.093533+0000","last_fullsized":"2024-03-11T22:50:56.093533+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:21:30.334268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094107+0000","last_change":"2024-03-11T22:50:26.900499+0000","last_active":"2024-03-11T22:50:56.094107+0000","last_peered":"2024-03-11T22:50:56.094107+0000","last_clean":"2024-03-11T22:50:56.094107+0000","last_became_active":"2024-03-11T22:50:26.899307+0000","last_became_peered":"2024-03-11T22:50:26.899307+0000","last_unstale":"2024-03-11T22:50:56.094107+0000","last_undegraded":"2024-03-11T22:50:56.094107+0000","last_fullsized":"2024-03-11T22:50:56.094107+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:28:18.008321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095776+0000","last_change":"2024-03-11T22:50:28.910462+0000","last_active":"2024-03-11T22:50:56.095776+0000","last_peered":"2024-03-11T22:50:56.095776+0000","last_clean":"2024-03-11T22:50:56.095776+0000","last_became_active":"2024-03-11T22:50:28.910111+0000","last_became_peered":"2024-03-11T22:50:28.910111+0000","last_unstale":"2024-03-11T22:50:56.095776+0000","last_undegraded":"2024-03-11T22:50:56.095776+0000","last_fullsized":"2024-03-11T22:50:56.095776+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:03:07.224641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952414+0000","last_change":"2024-03-11T22:50:30.921633+0000","last_active":"2024-03-11T22:51:16.952414+0000","last_peered":"2024-03-11T22:51:16.952414+0000","last_clean":"2024-03-11T22:51:16.952414+0000","last_became_active":"2024-03-11T22:50:30.920530+0000","last_became_peered":"2024-03-11T22:50:30.920530+0000","last_unstale":"2024-03-11T22:51:16.952414+0000","last_undegraded":"2024-03-11T22:51:16.952414+0000","last_fullsized":"2024-03-11T22:51:16.952414+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:46:50.900754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095127+0000","last_change":"2024-03-11T22:50:32.919726+0000","last_active":"2024-03-11T22:50:56.095127+0000","last_peered":"2024-03-11T22:50:56.095127+0000","last_clean":"2024-03-11T22:50:56.095127+0000","last_became_active":"2024-03-11T22:50:32.919104+0000","last_became_peered":"2024-03-11T22:50:32.919104+0000","last_unstale":"2024-03-11T22:50:56.095127+0000","last_undegraded":"2024-03-11T22:50:56.095127+0000","last_fullsized":"2024-03-11T22:50:56.095127+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:08:28.728474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096747+0000","last_change":"2024-03-11T22:50:26.903909+0000","last_active":"2024-03-11T22:50:56.096747+0000","last_peered":"2024-03-11T22:50:56.096747+0000","last_clean":"2024-03-11T22:50:56.096747+0000","last_became_active":"2024-03-11T22:50:26.903301+0000","last_became_peered":"2024-03-11T22:50:26.903301+0000","last_unstale":"2024-03-11T22:50:56.096747+0000","last_undegraded":"2024-03-11T22:50:56.096747+0000","last_fullsized":"2024-03-11T22:50:56.096747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:59:54.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093886+0000","last_change":"2024-03-11T22:50:28.910429+0000","last_active":"2024-03-11T22:50:56.093886+0000","last_peered":"2024-03-11T22:50:56.093886+0000","last_clean":"2024-03-11T22:50:56.093886+0000","last_became_active":"2024-03-11T22:50:28.909904+0000","last_became_peered":"2024-03-11T22:50:28.909904+0000","last_unstale":"2024-03-11T22:50:56.093886+0000","last_undegraded":"2024-03-11T22:50:56.093886+0000","last_fullsized":"2024-03-11T22:50:56.093886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:18.464087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094420+0000","last_change":"2024-03-11T22:50:30.921673+0000","last_active":"2024-03-11T22:50:56.094420+0000","last_peered":"2024-03-11T22:50:56.094420+0000","last_clean":"2024-03-11T22:50:56.094420+0000","last_became_active":"2024-03-11T22:50:30.920291+0000","last_became_peered":"2024-03-11T22:50:30.920291+0000","last_unstale":"2024-03-11T22:50:56.094420+0000","last_undegraded":"2024-03-11T22:50:56.094420+0000","last_fullsized":"2024-03-11T22:50:56.094420+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:53:05.596992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096443+0000","last_change":"2024-03-11T22:50:32.920184+0000","last_active":"2024-03-11T22:50:56.096443+0000","last_peered":"2024-03-11T22:50:56.096443+0000","last_clean":"2024-03-11T22:50:56.096443+0000","last_became_active":"2024-03-11T22:50:32.919815+0000","last_became_peered":"2024-03-11T22:50:32.919815+0000","last_unstale":"2024-03-11T22:50:56.096443+0000","last_undegraded":"2024-03-11T22:50:56.096443+0000","last_fullsized":"2024-03-11T22:50:56.096443+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:27:51.650662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952733+0000","last_change":"2024-03-11T22:50:30.917450+0000","last_active":"2024-03-11T22:51:16.952733+0000","last_peered":"2024-03-11T22:51:16.952733+0000","last_clean":"2024-03-11T22:51:16.952733+0000","last_became_active":"2024-03-11T22:50:30.917059+0000","last_became_peered":"2024-03-11T22:50:30.917059+0000","last_unstale":"2024-03-11T22:51:16.952733+0000","last_undegraded":"2024-03-11T22:51:16.952733+0000","last_fullsized":"2024-03-11T22:51:16.952733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:24:50.757893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093486+0000","last_change":"2024-03-11T22:50:26.904377+0000","last_active":"2024-03-11T22:50:56.093486+0000","last_peered":"2024-03-11T22:50:56.093486+0000","last_clean":"2024-03-11T22:50:56.093486+0000","last_became_active":"2024-03-11T22:50:26.903973+0000","last_became_peered":"2024-03-11T22:50:26.903973+0000","last_unstale":"2024-03-11T22:50:56.093486+0000","last_undegraded":"2024-03-11T22:50:56.093486+0000","last_fullsized":"2024-03-11T22:50:56.093486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:46.921090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094802+0000","last_change":"2024-03-11T22:50:28.909499+0000","last_active":"2024-03-11T22:50:56.094802+0000","last_peered":"2024-03-11T22:50:56.094802+0000","last_clean":"2024-03-11T22:50:56.094802+0000","last_became_active":"2024-03-11T22:50:28.909054+0000","last_became_peered":"2024-03-11T22:50:28.909054+0000","last_unstale":"2024-03-11T22:50:56.094802+0000","last_undegraded":"2024-03-11T22:50:56.094802+0000","last_fullsized":"2024-03-11T22:50:56.094802+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:10.604872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096187+0000","last_change":"2024-03-11T22:50:32.923767+0000","last_active":"2024-03-11T22:50:56.096187+0000","last_peered":"2024-03-11T22:50:56.096187+0000","last_clean":"2024-03-11T22:50:56.096187+0000","last_became_active":"2024-03-11T22:50:32.923508+0000","last_became_peered":"2024-03-11T22:50:32.923508+0000","last_unstale":"2024-03-11T22:50:56.096187+0000","last_undegraded":"2024-03-11T22:50:56.096187+0000","last_fullsized":"2024-03-11T22:50:56.096187+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:11:20.110632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094568+0000","last_change":"2024-03-11T22:50:30.921573+0000","last_active":"2024-03-11T22:50:56.094568+0000","last_peered":"2024-03-11T22:50:56.094568+0000","last_clean":"2024-03-11T22:50:56.094568+0000","last_became_active":"2024-03-11T22:50:30.920713+0000","last_became_peered":"2024-03-11T22:50:30.920713+0000","last_unstale":"2024-03-11T22:50:56.094568+0000","last_undegraded":"2024-03-11T22:50:56.094568+0000","last_fullsized":"2024-03-11T22:50:56.094568+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:51:16.389553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093839+0000","last_change":"2024-03-11T22:50:26.900505+0000","last_active":"2024-03-11T22:50:56.093839+0000","last_peered":"2024-03-11T22:50:56.093839+0000","last_clean":"2024-03-11T22:50:56.093839+0000","last_became_active":"2024-03-11T22:50:26.900196+0000","last_became_peered":"2024-03-11T22:50:26.900196+0000","last_unstale":"2024-03-11T22:50:56.093839+0000","last_undegraded":"2024-03-11T22:50:56.093839+0000","last_fullsized":"2024-03-11T22:50:56.093839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:16:38.760700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094625+0000","last_change":"2024-03-11T22:50:28.909392+0000","last_active":"2024-03-11T22:50:56.094625+0000","last_peered":"2024-03-11T22:50:56.094625+0000","last_clean":"2024-03-11T22:50:56.094625+0000","last_became_active":"2024-03-11T22:50:28.908984+0000","last_became_peered":"2024-03-11T22:50:28.908984+0000","last_unstale":"2024-03-11T22:50:56.094625+0000","last_undegraded":"2024-03-11T22:50:56.094625+0000","last_fullsized":"2024-03-11T22:50:56.094625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:25:31.458499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095888+0000","last_change":"2024-03-11T22:50:32.919346+0000","last_active":"2024-03-11T22:50:56.095888+0000","last_peered":"2024-03-11T22:50:56.095888+0000","last_clean":"2024-03-11T22:50:56.095888+0000","last_became_active":"2024-03-11T22:50:32.919019+0000","last_became_peered":"2024-03-11T22:50:32.919019+0000","last_unstale":"2024-03-11T22:50:56.095888+0000","last_undegraded":"2024-03-11T22:50:56.095888+0000","last_fullsized":"2024-03-11T22:50:56.095888+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:21.121389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095194+0000","last_change":"2024-03-11T22:50:26.899789+0000","last_active":"2024-03-11T22:50:56.095194+0000","last_peered":"2024-03-11T22:50:56.095194+0000","last_clean":"2024-03-11T22:50:56.095194+0000","last_became_active":"2024-03-11T22:50:26.899464+0000","last_became_peered":"2024-03-11T22:50:26.899464+0000","last_unstale":"2024-03-11T22:50:56.095194+0000","last_undegraded":"2024-03-11T22:50:56.095194+0000","last_fullsized":"2024-03-11T22:50:56.095194+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:23:47.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093743+0000","last_change":"2024-03-11T22:50:28.908081+0000","last_active":"2024-03-11T22:50:56.093743+0000","last_peered":"2024-03-11T22:50:56.093743+0000","last_clean":"2024-03-11T22:50:56.093743+0000","last_became_active":"2024-03-11T22:50:28.907753+0000","last_became_peered":"2024-03-11T22:50:28.907753+0000","last_unstale":"2024-03-11T22:50:56.093743+0000","last_undegraded":"2024-03-11T22:50:56.093743+0000","last_fullsized":"2024-03-11T22:50:56.093743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:21.170870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094617+0000","last_change":"2024-03-11T22:50:30.919101+0000","last_active":"2024-03-11T22:50:56.094617+0000","last_peered":"2024-03-11T22:50:56.094617+0000","last_clean":"2024-03-11T22:50:56.094617+0000","last_became_active":"2024-03-11T22:50:30.918124+0000","last_became_peered":"2024-03-11T22:50:30.918124+0000","last_unstale":"2024-03-11T22:50:56.094617+0000","last_undegraded":"2024-03-11T22:50:56.094617+0000","last_fullsized":"2024-03-11T22:50:56.094617+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:00.364123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092995+0000","last_change":"2024-03-11T22:50:32.920757+0000","last_active":"2024-03-11T22:50:56.092995+0000","last_peered":"2024-03-11T22:50:56.092995+0000","last_clean":"2024-03-11T22:50:56.092995+0000","last_became_active":"2024-03-11T22:50:32.919638+0000","last_became_peered":"2024-03-11T22:50:32.919638+0000","last_unstale":"2024-03-11T22:50:56.092995+0000","last_undegraded":"2024-03-11T22:50:56.092995+0000","last_fullsized":"2024-03-11T22:50:56.092995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:10:38.621924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:51:16.952741+0000","last_change":"2024-03-11T22:50:30.919406+0000","last_active":"2024-03-11T22:51:16.952741+0000","last_peered":"2024-03-11T22:51:16.952741+0000","last_clean":"2024-03-11T22:51:16.952741+0000","last_became_active":"2024-03-11T22:50:30.919023+0000","last_became_peered":"2024-03-11T22:50:30.919023+0000","last_unstale":"2024-03-11T22:51:16.952741+0000","last_undegraded":"2024-03-11T22:51:16.952741+0000","last_fullsized":"2024-03-11T22:51:16.952741+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:00:02.577888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095231+0000","last_change":"2024-03-11T22:50:26.899786+0000","last_active":"2024-03-11T22:50:56.095231+0000","last_peered":"2024-03-11T22:50:56.095231+0000","last_clean":"2024-03-11T22:50:56.095231+0000","last_became_active":"2024-03-11T22:50:26.899392+0000","last_became_peered":"2024-03-11T22:50:26.899392+0000","last_unstale":"2024-03-11T22:50:56.095231+0000","last_undegraded":"2024-03-11T22:50:56.095231+0000","last_fullsized":"2024-03-11T22:50:56.095231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:01:28.909407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094393+0000","last_change":"2024-03-11T22:50:28.908445+0000","last_active":"2024-03-11T22:50:56.094393+0000","last_peered":"2024-03-11T22:50:56.094393+0000","last_clean":"2024-03-11T22:50:56.094393+0000","last_became_active":"2024-03-11T22:50:28.907844+0000","last_became_peered":"2024-03-11T22:50:28.907844+0000","last_unstale":"2024-03-11T22:50:56.094393+0000","last_undegraded":"2024-03-11T22:50:56.094393+0000","last_fullsized":"2024-03-11T22:50:56.094393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:32:31.167181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094578+0000","last_change":"2024-03-11T22:50:32.919282+0000","last_active":"2024-03-11T22:50:56.094578+0000","last_peered":"2024-03-11T22:50:56.094578+0000","last_clean":"2024-03-11T22:50:56.094578+0000","last_became_active":"2024-03-11T22:50:32.918901+0000","last_became_peered":"2024-03-11T22:50:32.918901+0000","last_unstale":"2024-03-11T22:50:56.094578+0000","last_undegraded":"2024-03-11T22:50:56.094578+0000","last_fullsized":"2024-03-11T22:50:56.094578+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:39:58.417112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.095748+0000","last_change":"2024-03-11T22:50:26.902721+0000","last_active":"2024-03-11T22:50:56.095748+0000","last_peered":"2024-03-11T22:50:56.095748+0000","last_clean":"2024-03-11T22:50:56.095748+0000","last_became_active":"2024-03-11T22:50:26.901836+0000","last_became_peered":"2024-03-11T22:50:26.901836+0000","last_unstale":"2024-03-11T22:50:56.095748+0000","last_undegraded":"2024-03-11T22:50:56.095748+0000","last_fullsized":"2024-03-11T22:50:56.095748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:44:17.145868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094797+0000","last_change":"2024-03-11T22:50:28.909865+0000","last_active":"2024-03-11T22:50:56.094797+0000","last_peered":"2024-03-11T22:50:56.094797+0000","last_clean":"2024-03-11T22:50:56.094797+0000","last_became_active":"2024-03-11T22:50:28.908723+0000","last_became_peered":"2024-03-11T22:50:28.908723+0000","last_unstale":"2024-03-11T22:50:56.094797+0000","last_undegraded":"2024-03-11T22:50:56.094797+0000","last_fullsized":"2024-03-11T22:50:56.094797+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:22:47.663612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093886+0000","last_change":"2024-03-11T22:50:30.912723+0000","last_active":"2024-03-11T22:50:56.093886+0000","last_peered":"2024-03-11T22:50:56.093886+0000","last_clean":"2024-03-11T22:50:56.093886+0000","last_became_active":"2024-03-11T22:50:30.912299+0000","last_became_peered":"2024-03-11T22:50:30.912299+0000","last_unstale":"2024-03-11T22:50:56.093886+0000","last_undegraded":"2024-03-11T22:50:56.093886+0000","last_fullsized":"2024-03-11T22:50:56.093886+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:57.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.092438+0000","last_change":"2024-03-11T22:50:32.918484+0000","last_active":"2024-03-11T22:50:56.092438+0000","last_peered":"2024-03-11T22:50:56.092438+0000","last_clean":"2024-03-11T22:50:56.092438+0000","last_became_active":"2024-03-11T22:50:32.918096+0000","last_became_peered":"2024-03-11T22:50:32.918096+0000","last_unstale":"2024-03-11T22:50:56.092438+0000","last_undegraded":"2024-03-11T22:50:56.092438+0000","last_fullsized":"2024-03-11T22:50:56.092438+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:35:45.888334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096163+0000","last_change":"2024-03-11T22:50:26.903957+0000","last_active":"2024-03-11T22:50:56.096163+0000","last_peered":"2024-03-11T22:50:56.096163+0000","last_clean":"2024-03-11T22:50:56.096163+0000","last_became_active":"2024-03-11T22:50:26.903316+0000","last_became_peered":"2024-03-11T22:50:26.903316+0000","last_unstale":"2024-03-11T22:50:56.096163+0000","last_undegraded":"2024-03-11T22:50:56.096163+0000","last_fullsized":"2024-03-11T22:50:56.096163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:36:42.965662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094931+0000","last_change":"2024-03-11T22:50:28.911567+0000","last_active":"2024-03-11T22:50:56.094931+0000","last_peered":"2024-03-11T22:50:56.094931+0000","last_clean":"2024-03-11T22:50:56.094931+0000","last_became_active":"2024-03-11T22:50:28.910618+0000","last_became_peered":"2024-03-11T22:50:28.910618+0000","last_unstale":"2024-03-11T22:50:56.094931+0000","last_undegraded":"2024-03-11T22:50:56.094931+0000","last_fullsized":"2024-03-11T22:50:56.094931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:38:09.997753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093132+0000","last_change":"2024-03-11T22:50:30.914252+0000","last_active":"2024-03-11T22:50:56.093132+0000","last_peered":"2024-03-11T22:50:56.093132+0000","last_clean":"2024-03-11T22:50:56.093132+0000","last_became_active":"2024-03-11T22:50:30.913872+0000","last_became_peered":"2024-03-11T22:50:30.913872+0000","last_unstale":"2024-03-11T22:50:56.093132+0000","last_undegraded":"2024-03-11T22:50:56.093132+0000","last_fullsized":"2024-03-11T22:50:56.093132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:24:15.526871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093392+0000","last_change":"2024-03-11T22:50:32.919850+0000","last_active":"2024-03-11T22:50:56.093392+0000","last_peered":"2024-03-11T22:50:56.093392+0000","last_clean":"2024-03-11T22:50:56.093392+0000","last_became_active":"2024-03-11T22:50:32.919493+0000","last_became_peered":"2024-03-11T22:50:32.919493+0000","last_unstale":"2024-03-11T22:50:56.093392+0000","last_undegraded":"2024-03-11T22:50:56.093392+0000","last_fullsized":"2024-03-11T22:50:56.093392+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:31.903244+0000","last_clean_scrub_stamp":"2024-03-11T22:50:31.903244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:27:41.693416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.093787+0000","last_change":"2024-03-11T22:50:26.900752+0000","last_active":"2024-03-11T22:50:56.093787+0000","last_peered":"2024-03-11T22:50:56.093787+0000","last_clean":"2024-03-11T22:50:56.093787+0000","last_became_active":"2024-03-11T22:50:26.900230+0000","last_became_peered":"2024-03-11T22:50:26.900230+0000","last_unstale":"2024-03-11T22:50:56.093787+0000","last_undegraded":"2024-03-11T22:50:56.093787+0000","last_fullsized":"2024-03-11T22:50:56.093787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:25.879730+0000","last_clean_scrub_stamp":"2024-03-11T22:50:25.879730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:23:55.485115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.096780+0000","last_change":"2024-03-11T22:50:28.910491+0000","last_active":"2024-03-11T22:50:56.096780+0000","last_peered":"2024-03-11T22:50:56.096780+0000","last_clean":"2024-03-11T22:50:56.096780+0000","last_became_active":"2024-03-11T22:50:28.909943+0000","last_became_peered":"2024-03-11T22:50:28.909943+0000","last_unstale":"2024-03-11T22:50:56.096780+0000","last_undegraded":"2024-03-11T22:50:56.096780+0000","last_fullsized":"2024-03-11T22:50:56.096780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:27.890370+0000","last_clean_scrub_stamp":"2024-03-11T22:50:27.890370+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:59.210238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T22:50:56.094898+0000","last_change":"2024-03-11T22:50:30.920607+0000","last_active":"2024-03-11T22:50:56.094898+0000","last_peered":"2024-03-11T22:50:56.094898+0000","last_clean":"2024-03-11T22:50:56.094898+0000","last_became_active":"2024-03-11T22:50:30.920259+0000","last_became_peered":"2024-03-11T22:50:30.920259+0000","last_unstale":"2024-03-11T22:50:56.094898+0000","last_undegraded":"2024-03-11T22:50:56.094898+0000","last_fullsized":"2024-03-11T22:50:56.094898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:50:29.900071+0000","last_clean_scrub_stamp":"2024-03-11T22:50:29.900071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:48:00.145519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411345,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36376,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704680,"statfs":{"total":95990841344,"available":95953592320,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708840,"statfs":{"total":95990841344,"available":95957852160,"internally_reserved":0,"allocated":1060864,"data_stored":703166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:51:27.569 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:51:27.569 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T22:51:27.569 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T22:51:27.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T22:51:27.811 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:27.820 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:51:27 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:51:27.738Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002251541s 2024-03-11T22:51:28.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:27 smithi052 ceph-mon[25824]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:28.211 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:27 smithi052 ceph-mon[25824]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:51:28.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:27 smithi116 ceph-mon[27552]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:28.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:27 smithi116 ceph-mon[27552]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:51:28.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:27 smithi152 ceph-mon[27791]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:28.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:27 smithi152 ceph-mon[27791]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:51:29.808 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-11T22:51:29.808 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:51:30.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:29 smithi052 ceph-mon[25824]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:30.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:29 smithi052 ceph-mon[25824]: from='client.? 172.21.15.52:0/3017805148' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:51:30.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:29 smithi116 ceph-mon[27552]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:30.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:29 smithi116 ceph-mon[27552]: from='client.? 172.21.15.52:0/3017805148' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:51:30.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:29 smithi152 ceph-mon[27791]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:30.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:29 smithi152 ceph-mon[27791]: from='client.? 172.21.15.52:0/3017805148' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:51:30.431 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T22:51:30.431 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T22:51:30.432 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:51:30.441 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi052.front.sepia.ceph.com 2024-03-11T22:51:30.442 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T22:51:30.681 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:31.762 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:31 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:51:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:51:31.978 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:31 smithi116 ceph-mon[27552]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:32.069 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:31 smithi052 ceph-mon[25824]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:32.322 INFO:teuthology.orchestra.run.smithi052.stdout:167 167 2024-03-11T22:51:32.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:31 smithi152 ceph-mon[27791]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:32.979 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T22:51:33.219 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:33.784 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 podman[38114]: 2024-03-11 22:51:33.526380923 +0000 UTC m=+0.120305394 container create 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, architecture=x86_64, release=239.1651231664, distribution-scope=public, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, version=8.5, vendor=Red Hat, Inc.) 2024-03-11T22:51:33.784 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 podman[38114]: 2024-03-11 22:51:33.778576912 +0000 UTC m=+0.372501391 container init 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, name=ubi8, version=8.5, release=239.1651231664, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-type=git) 2024-03-11T22:51:34.035 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-mon[27552]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:34.035 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 podman[38114]: 2024-03-11 22:51:33.784320986 +0000 UTC m=+0.378245466 container start 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a, vendor=Red Hat, Inc., io.buildah.version=1.32.0, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, release=239.1651231664, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , version=8.5, description=Ceph Grafana Container, name=ubi8, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 bash[38114]: 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 systemd[1]: Started Ceph grafana.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.93408766Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934271567Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934288569Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934295265Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934301067Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934308138Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934314296Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934320376Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934326288Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T22:51:34.036 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934332167Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934337561Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934342945Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934348575Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=settings t=2024-03-11T22:51:33.934354214Z level=info msg="App mode production" 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=sqlstore t=2024-03-11T22:51:33.934404339Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=sqlstore t=2024-03-11T22:51:33.934421984Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T22:51:34.037 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.935267405Z level=info msg="Starting DB migrations" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.935344422Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.935859283Z level=info msg="Executing migration" id="create user table" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.936348765Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.93721121Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.93805022Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.938850326Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.939701829Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.940601064Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.941463871Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.94231256Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.945185819Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.947992162Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T22:51:34.038 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.950694801Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.953378796Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.953811098Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.956543698Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.959258169Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.962063045Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.96246187Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.965140223Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.968056425Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.973577053Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.97415085Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.976777943Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.979386526Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.982288432Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T22:51:34.039 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.984904037Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.987838762Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.990487853Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.993196065Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.995903732Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:33 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:33.998570719Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.001173664Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.003786519Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.006424034Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.009175857Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.012190487Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.014537532Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.017189126Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T22:51:34.040 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.01779961Z level=info msg="Executing migration" id="create star table" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.020657567Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.023072639Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.02568314Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.028285788Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.030853771Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.031683566Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.032130906Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T22:51:34.041 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.032963106Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T22:51:34.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:33 smithi052 ceph-mon[25824]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:34.286 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.033886469Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.034554955Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.034867801Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.035299596Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.035732794Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.036699613Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.037363535Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.037844466Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.038675152Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.039560768Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.040323987Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.040764764Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T22:51:34.287 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.041605291Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T22:51:34.288 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.042409739Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T22:51:34.288 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.043343019Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T22:51:34.288 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.043758949Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T22:51:34.288 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.044642242Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T22:51:34.288 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.045517549Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T22:51:34.288 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.046365739Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.046865782Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.047722796Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.048707857Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.049433682Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.049772814Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.052333834Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.055051954Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T22:51:34.289 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.055631778Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.056165007Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.058939683Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.061529139Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.063986199Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.066530962Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.069292458Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.072100862Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.074732415Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.075690949Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.07839102Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.080994273Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.083862134Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T22:51:34.290 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.084396046Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.08704736Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.087579193Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.089640551Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.090007893Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.090487815Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.090947415Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.091795117Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.092693813Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.093260303Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.09368889Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.094492822Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.095487197Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.096580527Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T22:51:34.291 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.097039447Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.098008662Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.098650711Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.099447323Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.099995064Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.100541095Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.101439353Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.102019117Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.102550451Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.103400199Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.103731255Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.104289438Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.105155551Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.106068636Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T22:51:34.292 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.106852198Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.107258751Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.10810855Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.109005536Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.11031096Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.110728097Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.111669042Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.112471542Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.113543799Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.114351712Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.114780685Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.115208534Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.115637291Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.115949031Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T22:51:34.293 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.116308821Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.116581479Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.117157983Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.117698011Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.11799467Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.11858517Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.119137158Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.119529206Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.119877559Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.120281933Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.120713544Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.121162853Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.121595454Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.121868916Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T22:51:34.294 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.122195748Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.122722828Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.123296109Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.123613015Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.124001675Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.12445346Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.124735012Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.125144228Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.125582781Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.126186267Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.126517915Z level=info msg="Executing migration" id="create session table" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.126962751Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.127269409Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T22:51:34.295 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.127558335Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.127962167Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.128362603Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.128635386Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.128901102Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.129233256Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.129515411Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.129906126Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.130221875Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.130797914Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.131131377Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.131731633Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.132323121Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.132628719Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.133098215Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T22:51:34.296 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.13357227Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.134393707Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.137188794Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.139983746Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.140565064Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.141011481Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.141883624Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.142312645Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.143764438Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.145055508Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T22:51:34.297 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.147741214Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.148098262Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.148464275Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.148846508Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.149484807Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.150148181Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.150787533Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.151453471Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.151928286Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T22:51:34.298 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.152264862Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.152585472Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.152983748Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.153429039Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.153821701Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.154255055Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.157003736Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.159724142Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.162458503Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.165224273Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.168086088Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.170866523Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.173783968Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.176377571Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.179268738Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T22:51:34.299 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.182035999Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.184991171Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.187777409Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.190412455Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.192984257Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.195728438Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.198344211Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.200909342Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.20346093Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.206175136Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.208578331Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.211313978Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.214036773Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.216865157Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.217408377Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T22:51:34.300 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.220062857Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.222833097Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.225639Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.228443864Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.23114376Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.233803057Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.23652242Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.239296628Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.242237954Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.244906612Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.245398673Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.248117354Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.250758173Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.253500083Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T22:51:34.301 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.25392145Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.256944418Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.259412363Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.262309004Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.265000813Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.267515048Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.270176066Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.272761259Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.273285111Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.27603083Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.278750747Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.281382667Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.281881788Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T22:51:34.302 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.284564412Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T22:51:34.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:33 smithi152 ceph-mon[27791]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:34.538 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.287247088Z level=info msg="Executing migration" id="create team table" 2024-03-11T22:51:34.538 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.289819155Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T22:51:34.538 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.29032957Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T22:51:34.538 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.292849041Z level=info msg="Executing migration" id="create team member table" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.295442266Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.298185892Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.298624293Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.299393742Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.300182313Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.30099648Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.301713438Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.30248735Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.303288209Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.304116484Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.304552657Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T22:51:34.539 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.305396074Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.30638113Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.307082408Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.307898338Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.309298506Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.309840127Z level=info msg="Executing migration" id="create tag table" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.310687565Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.311496109Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.312332538Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.313306742Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.314179372Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.316192779Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.317035664Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.317901372Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T22:51:34.540 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.318749222Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.319714588Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.320441291Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.321462881Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.32219267Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.323037116Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.324028283Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.324927406Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T22:51:34.541 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.325778176Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.32663886Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.327701733Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.328368401Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.329419544Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.332220605Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.334966112Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.337840673Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.340549003Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.343295737Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.346080332Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.348821166Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T22:51:34.543 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.351705809Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.354416356Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.357229592Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.360103035Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.362609902Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.365453425Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.368216441Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.370794016Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.373699128Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.37623332Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.378988565Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.381656672Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.384309783Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.386861071Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T22:51:34.544 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.390029717Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.392191941Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.395086003Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.397596984Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.400294098Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.403074442Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.40562187Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.408303106Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.411025891Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.413689195Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.416535075Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.419055085Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.422459241Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.42586527Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T22:51:34.545 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.426861985Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.427570587Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.428432684Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.42928366Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.43013459Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.431094487Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.431768251Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.432636188Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.433513435Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.434389886Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.435200912Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.436047184Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.436879169Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.437685193Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T22:51:34.546 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.43853011Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.439489066Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.440453852Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.441301688Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.442299576Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.443023851Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.443944355Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.444390682Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.444656046Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.445530855Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.446418212Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.447299647Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.448132344Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.448958842Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T22:51:34.547 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.449801989Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.450577998Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.451485213Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.45233971Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.453292529Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.454121374Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.455032373Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.455860186Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.456711915Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.457671627Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.458506169Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.460270393Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.460828083Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.461702176Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T22:51:34.548 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.462550658Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.463360502Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.463806337Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.463878904Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.464618861Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.465476759Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.466366464Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.466486443Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.466591105Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.46726709Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.46812331Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.469029165Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.469894476Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.470746452Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T22:51:34.549 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.471597361Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.472434338Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.473294511Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.47416038Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.475033218Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.479308408Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.482146289Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.484957833Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.48915689Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.493382424Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.495934049Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.496384789Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.49669246Z level=info msg="Executing migration" id="create permission table" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.497086504Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T22:51:34.550 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.497508761Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.497936801Z level=info msg="Executing migration" id="create role table" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.498342568Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.499243895Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.500147377Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.500596169Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.501360269Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.501772878Z level=info msg="Executing migration" id="create team role table" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.502160145Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.502579877Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.503033693Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.503450461Z level=info msg="Executing migration" id="create user role table" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.503902216Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.504327103Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T22:51:34.551 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.504741161Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.505185149Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.505595694Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.508344333Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.508761506Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.509756902Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.510251912Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.512958461Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.513386592Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.513778201Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.51416232Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.514586509Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.515544837Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.515944062Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.516366145Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T22:51:34.552 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.516624522Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.517029115Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.517501253Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.517836838Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.518142692Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.518486087Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.518863978Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.519301054Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.520275592Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.520576302Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.520970946Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.521384729Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.521789136Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.522756834Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T22:51:34.553 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.523178181Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.52597349Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.52606699Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.527334842Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.527414038Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.527692495Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.530598173Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.533370399Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:51:34.554 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.53617581Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.538779175Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.541617185Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.544363426Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.547249917Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.550032936Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.552648145Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:51:34.937 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.555332886Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.558033688Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.561144808Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.56383725Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.566314962Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.568922257Z level=info msg="Executing migration" id="add share column" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.5716927Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.574418649Z level=info msg="Executing migration" id="create file table" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.577077153Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.579861062Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.582596268Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.583034132Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.585702109Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.588394783Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T22:51:34.938 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.591159782Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.593736494Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.596419024Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.599085414Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.599942147Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.600789657Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.601600584Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.602389549Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.603247572Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.604274716Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.605468593Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.606285242Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.616196523Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.617036128Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.617858733Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T22:51:34.939 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.621325675Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.622096241Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.62297996Z level=info msg="Executing migration" id="create folder table" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.623466262Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.624244711Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.62509484Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.62598562Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=migrator t=2024-03-11T22:51:34.626836822Z level=info msg="migrations completed" performed=485 skipped=0 duration=691.515358ms 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=sqlstore t=2024-03-11T22:51:34.627248891Z level=info msg="Created default organization" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=plugin.loader t=2024-03-11T22:51:34.683957111Z level=info msg="Plugin registered" pluginID=input 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=plugin.loader t=2024-03-11T22:51:34.713147473Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=plugin.loader t=2024-03-11T22:51:34.713335636Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=secrets t=2024-03-11T22:51:34.713372547Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=query_data t=2024-03-11T22:51:34.715523129Z level=info msg="Query Service initialization" 2024-03-11T22:51:34.940 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=live.push_http t=2024-03-11T22:51:34.717941538Z level=info msg="Live Push Gateway initialization" 2024-03-11T22:51:35.078 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:34 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.079 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:34 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.079 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:34 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.079 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:34 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:34 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:34 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:34 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:34 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:34 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:35.330 INFO:teuthology.orchestra.run.smithi052.stdout:Backend: cephadm 2024-03-11T22:51:35.330 INFO:teuthology.orchestra.run.smithi052.stdout:Available: Yes 2024-03-11T22:51:35.330 INFO:teuthology.orchestra.run.smithi052.stdout:Paused: No 2024-03-11T22:51:35.331 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:35 smithi052 systemd[1]: Starting Ceph node-exporter.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:51:35.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=infra.usagestats.collector t=2024-03-11T22:51:35.367066799Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T22:51:35.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=provisioning.datasources t=2024-03-11T22:51:35.368090043Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T22:51:35.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=provisioning.datasources t=2024-03-11T22:51:35.37898401Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T22:51:35.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=provisioning.alerting t=2024-03-11T22:51:35.389617934Z level=info msg="starting to provision alerting" 2024-03-11T22:51:35.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=provisioning.alerting t=2024-03-11T22:51:35.389635778Z level=info msg="finished to provision alerting" 2024-03-11T22:51:35.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=ngalert.state.manager t=2024-03-11T22:51:35.389743931Z level=info msg="Warming state cache for startup" 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=grafanaStorageLogger t=2024-03-11T22:51:35.389833866Z level=info msg="storage starting" 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=ngalert.state.manager t=2024-03-11T22:51:35.389910341Z level=info msg="State cache has been initialized" states=0 duration=165.5µs 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=ticker t=2024-03-11T22:51:35.390680181Z level=info msg=starting first_tick=2024-03-11T22:51:40Z 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=ngalert.multiorg.alertmanager t=2024-03-11T22:51:35.390699898Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=http.server t=2024-03-11T22:51:35.391780556Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=sqlstore.transactions t=2024-03-11T22:51:35.405909132Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=sqlstore.transactions t=2024-03-11T22:51:35.407518655Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:51:35.738 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a[38129]: logger=sqlstore.transactions t=2024-03-11T22:51:35.418995148Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-11T22:51:35.945 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T22:51:36.043 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-mon[27552]: Deploying daemon node-exporter.a on smithi052 2024-03-11T22:51:36.043 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:35 smithi116 ceph-mon[27552]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:36.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:35 smithi152 ceph-mon[27791]: Deploying daemon node-exporter.a on smithi052 2024-03-11T22:51:36.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:35 smithi152 ceph-mon[27791]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:36.202 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:36.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:35 smithi052 ceph-mon[25824]: Deploying daemon node-exporter.a on smithi052 2024-03-11T22:51:36.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:35 smithi052 ceph-mon[25824]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:36.321 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:36 smithi052 bash[50145]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:51:36.660 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:36 smithi052 bash[50145]: Getting image source signatures 2024-03-11T22:51:36.660 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:36 smithi052 bash[50145]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:51:36.661 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:36 smithi052 bash[50145]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:51:36.661 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:36 smithi052 bash[50145]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:51:37.193 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:36 smithi116 ceph-mon[27552]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:37.193 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:36 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:37.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:36 smithi052 ceph-mon[25824]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:37.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:36 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:37.328 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:36 smithi152 ceph-mon[27791]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:37.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:36 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:37.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:37 smithi116 ceph-mon[27552]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:38.155 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:37 smithi052 ceph-mon[25824]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:38.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:37 smithi152 ceph-mon[27791]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:38.525 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:38 smithi052 bash[50145]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:51:38.820 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:38 smithi052 bash[50145]: Writing manifest to image destination 2024-03-11T22:51:39.571 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:39 smithi052 podman[50145]: 2024-03-11 22:51:39.108135876 +0000 UTC m=+3.084257073 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:51:40.189 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:39 smithi152 ceph-mon[27791]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:40.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:39 smithi116 ceph-mon[27552]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:40.258 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:39 smithi052 ceph-mon[25824]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:agent.smithi052 smithi052 running 51s ago 3m - - 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:agent.smithi116 smithi116 running 63s ago 3m - - 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:agent.smithi152 smithi152 running 64s ago 3m - - 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager.a smithi052 *:9093,9094 starting - - - - 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:grafana.a smithi116 *:3000 starting - - - - 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.x smithi116 *:8443,8765 running (2m) 63s ago 2m 451M - 19.0.0-1607-g94a53970 a6477a474d71 72598752c6e6 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.y smithi052 *:9283,8765 running (4m) 51s ago 4m 515M - 19.0.0-1607-g94a53970 a6477a474d71 afc4e2cd7338 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:mgr.z smithi152 *:8443,8765 running (2m) 64s ago 2m 451M - 19.0.0-1607-g94a53970 a6477a474d71 6265e2d4f5ac 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:mon.a smithi052 running (4m) 51s ago 4m 46.9M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d0af2840eadd 2024-03-11T22:51:40.374 INFO:teuthology.orchestra.run.smithi052.stdout:mon.b smithi116 running (3m) 63s ago 3m 40.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 8a1c8d1a55fd 2024-03-11T22:51:40.375 INFO:teuthology.orchestra.run.smithi052.stdout:mon.c smithi152 running (3m) 64s ago 3m 41.5M 2048M 19.0.0-1607-g94a53970 a6477a474d71 074b8af07f2c 2024-03-11T22:51:40.375 INFO:teuthology.orchestra.run.smithi052.stdout:osd.0 smithi052 running (2m) 51s ago 2m 57.7M 4096M 19.0.0-1607-g94a53970 a6477a474d71 be36f578cff2 2024-03-11T22:51:40.375 INFO:teuthology.orchestra.run.smithi052.stdout:osd.1 smithi116 running (2m) 63s ago 2m 60.3M 4096M 19.0.0-1607-g94a53970 a6477a474d71 45ce9126aa80 2024-03-11T22:51:40.375 INFO:teuthology.orchestra.run.smithi052.stdout:osd.2 smithi152 running (94s) 64s ago 93s 48.6M 4096M 19.0.0-1607-g94a53970 a6477a474d71 7c6f38f4bcbc 2024-03-11T22:51:40.375 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus.a smithi116 *:9095 starting - - - - 2024-03-11T22:51:40.375 INFO:teuthology.orchestra.run.smithi052.stdout:rgw.foo.a smithi052 *:80 running (75s) 51s ago 75s 74.6M - 19.0.0-1607-g94a53970 a6477a474d71 d5b67d408122 2024-03-11T22:51:40.512 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 podman[50145]: 2024-03-11 22:51:40.254108899 +0000 UTC m=+4.230230090 container create 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:51:40.512 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 podman[50145]: 2024-03-11 22:51:40.498214144 +0000 UTC m=+4.474335341 container init 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:51:40.512 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 podman[50145]: 2024-03-11 22:51:40.503696563 +0000 UTC m=+4.479817765 container start 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.507Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.507Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.507Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.507Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:51:40.821 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:51:40.822 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:51:40.823 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.508Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:51:40.824 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:51:40.825 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:51:40.826 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a[50425]: ts=2024-03-11T22:51:40.509Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:51:40.826 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 bash[50145]: 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 2024-03-11T22:51:40.826 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:51:40 smithi052 systemd[1]: Started Ceph node-exporter.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:51:41.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:51:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: Deploying daemon node-exporter.b on smithi116 2024-03-11T22:51:41.785 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:41 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:41.847 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:41.847 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:41.847 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.848 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.848 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.848 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: Deploying daemon node-exporter.b on smithi116 2024-03-11T22:51:41.848 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:41 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: Deploying daemon node-exporter.b on smithi116 2024-03-11T22:51:41.963 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:41 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:42.237 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:41 smithi116 bash[39172]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:51:42.237 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:42.987 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:42 smithi116 bash[39172]: Getting image source signatures 2024-03-11T22:51:42.987 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:42 smithi116 bash[39172]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:51:42.987 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:42 smithi116 bash[39172]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:51:42.987 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:42 smithi116 bash[39172]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:51:43.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:43 smithi116 ceph-mon[27552]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:44.048 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:43 smithi052 ceph-mon[25824]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:44.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:43 smithi152 ceph-mon[27791]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:44.810 INFO:teuthology.orchestra.run.smithi052.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T22:51:44.810 INFO:teuthology.orchestra.run.smithi052.stdout:agent 3/3 69s ago 48s * 2024-03-11T22:51:44.810 INFO:teuthology.orchestra.run.smithi052.stdout:alertmanager ?:9093,9094 0/1 - 74s smithi052=a;count:1 2024-03-11T22:51:44.810 INFO:teuthology.orchestra.run.smithi052.stdout:grafana ?:3000 0/1 - 71s smithi116=a;count:1 2024-03-11T22:51:44.810 INFO:teuthology.orchestra.run.smithi052.stdout:mgr 3/3 69s ago 2m smithi052=y;smithi116=x;smithi152=z;count:3 2024-03-11T22:51:44.811 INFO:teuthology.orchestra.run.smithi052.stdout:mon 3/3 69s ago 3m smithi052:172.21.15.52=a;smithi116:172.21.15.116=b;smithi152:172.21.15.152=c;count:3 2024-03-11T22:51:44.811 INFO:teuthology.orchestra.run.smithi052.stdout:node-exporter ?:9100 0/3 - 77s smithi052=a;smithi116=b;smithi152=c;count:3 2024-03-11T22:51:44.811 INFO:teuthology.orchestra.run.smithi052.stdout:osd 3 69s ago - 2024-03-11T22:51:44.811 INFO:teuthology.orchestra.run.smithi052.stdout:prometheus ?:9095 0/1 - 79s smithi116=a;count:1 2024-03-11T22:51:44.811 INFO:teuthology.orchestra.run.smithi052.stdout:rgw.foo.a ?:80 1/1 56s ago 79s smithi052=foo.a;count:1 2024-03-11T22:51:45.629 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T22:51:45.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:45 smithi052 ceph-mon[25824]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:45.947 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:45 smithi052 ceph-mon[25824]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:45.982 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:45.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:45 smithi116 ceph-mon[27552]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:45.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:45 smithi116 ceph-mon[27552]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:46.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:45 smithi152 ceph-mon[27791]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:46.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:45 smithi152 ceph-mon[27791]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:46.486 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 bash[39172]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:51:46.487 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 bash[39172]: Writing manifest to image destination 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 podman[39172]: 2024-03-11 22:51:46.513757987 +0000 UTC m=+4.764320251 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 podman[39172]: 2024-03-11 22:51:46.604153757 +0000 UTC m=+4.854716053 container create 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 podman[39172]: 2024-03-11 22:51:46.822450808 +0000 UTC m=+5.073013088 container init 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 podman[39172]: 2024-03-11 22:51:46.828274483 +0000 UTC m=+5.078836739 container start 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.832Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.832Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.832Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.832Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:51:46.919 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:51:46.920 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:51:46.921 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:51:46.922 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b[39221]: ts=2024-03-11T22:51:46.833Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:51:46.923 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 bash[39172]: 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 2024-03-11T22:51:47.237 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:51:46 smithi116 systemd[1]: Started Ceph node-exporter.b for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:51:47.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:47 smithi152 ceph-mon[27791]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:47.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:47 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:47.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:47 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:47.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:47 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:HOST ADDR LABELS STATUS 2024-03-11T22:51:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 172.21.15.52 2024-03-11T22:51:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 172.21.15.116 2024-03-11T22:51:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 172.21.15.152 2024-03-11T22:51:47.870 INFO:teuthology.orchestra.run.smithi052.stdout:3 hosts in cluster 2024-03-11T22:51:47.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:47 smithi116 ceph-mon[27552]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:47.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:47 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:47.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:47 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:47.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:47 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:48.062 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:47 smithi052 ceph-mon[25824]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:48.062 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:47 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:48.062 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:47 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:48.062 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:47 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:48.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:47 smithi152 systemd[1]: Starting Ceph node-exporter.c for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:51:48.579 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:48 smithi152 bash[39216]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:51:48.753 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T22:51:48.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:48 smithi116 ceph-mon[27552]: Deploying daemon node-exporter.c on smithi152 2024-03-11T22:51:48.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:48 smithi116 ceph-mon[27552]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:49.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:48 smithi052 ceph-mon[25824]: Deploying daemon node-exporter.c on smithi152 2024-03-11T22:51:49.000 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:48 smithi052 ceph-mon[25824]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:49.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:48 smithi152 ceph-mon[27791]: Deploying daemon node-exporter.c on smithi152 2024-03-11T22:51:49.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:48 smithi152 ceph-mon[27791]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:49.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:48 smithi152 bash[39216]: Getting image source signatures 2024-03-11T22:51:49.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:48 smithi152 bash[39216]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:51:49.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:48 smithi152 bash[39216]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:51:49.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:48 smithi152 bash[39216]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:51:49.991 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:49 smithi116 ceph-mon[27552]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:50.000 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:50.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:49 smithi052 ceph-mon[25824]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:50.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:49 smithi152 ceph-mon[27791]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:51.579 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 bash[39216]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:51:51.579 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 bash[39216]: Writing manifest to image destination 2024-03-11T22:51:51.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:51 smithi052 ceph-mon[25824]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:51.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:51:51 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:51:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:51:51.963 INFO:teuthology.orchestra.run.smithi052.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513010N480DGN 447G No 83s ago Has a FileSystem, LVM detected 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme1n1 ssd Linux_b6ba118092571df5e527 89.4G Yes 83s ago 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme2n1 ssd Linux_dc2184990ddde91cdd34 89.4G Yes 83s ago 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme3n1 ssd Linux_19feec2ada5b97b59cc7 89.4G Yes 83s ago 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/nvme4n1 ssd Linux_3458ba7a8a84a78a5817 89.4G No 83s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HRYS 931G No 83s ago Has partitions 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 75s ago Has a FileSystem, LVM detected 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 /dev/nvme1n1 ssd Linux_99d57a0f0300ab9bcc18 89.4G Yes 75s ago 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 /dev/nvme2n1 ssd Linux_ee4d1a537ce9016e5459 89.4G Yes 75s ago 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 /dev/nvme3n1 ssd Linux_cb9cb1c9415af4464ef7 89.4G Yes 75s ago 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 /dev/nvme4n1 ssd Linux_8d2640b8d59fb4b30626 89.4G No 75s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:51:51.964 INFO:teuthology.orchestra.run.smithi052.stdout:smithi116 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5QJB5 931G No 75s ago Has partitions 2024-03-11T22:51:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 96s ago Has a FileSystem, LVM detected 2024-03-11T22:51:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 /dev/nvme1n1 ssd Linux_310f4811b8f1d6273564 89.4G Yes 96s ago 2024-03-11T22:51:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 /dev/nvme2n1 ssd Linux_1718228592a696eb2c5e 89.4G Yes 96s ago 2024-03-11T22:51:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 /dev/nvme3n1 ssd Linux_f20c140db9425dfdb8e7 89.4G Yes 96s ago 2024-03-11T22:51:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 /dev/nvme4n1 ssd Linux_bd9aa492a179a6ea3c4e 89.4G No 96s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:51:51.965 INFO:teuthology.orchestra.run.smithi052.stdout:smithi152 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XEKT 931G No 96s ago Has partitions 2024-03-11T22:51:51.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:51 smithi116 ceph-mon[27552]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:52.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-mon[27791]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 podman[39216]: 2024-03-11 22:51:51.580025868 +0000 UTC m=+3.557853020 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 podman[39216]: 2024-03-11 22:51:51.669972574 +0000 UTC m=+3.647799728 container create ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 podman[39216]: 2024-03-11 22:51:51.888791843 +0000 UTC m=+3.866619009 container init ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 podman[39216]: 2024-03-11 22:51:51.894218317 +0000 UTC m=+3.872045476 container start ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.898Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.898Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.898Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.898Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:51:52.003 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:51:52.004 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:51:52.005 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:51:52.006 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:51:52.006 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:51:52.006 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:51:52.006 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:51:52.006 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:51:52.006 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:51:52.010 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:51:52.010 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c[39276]: ts=2024-03-11T22:51:51.899Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:51:52.011 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:51 smithi152 bash[39216]: ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 2024-03-11T22:51:52.329 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:51:52 smithi152 systemd[1]: Started Ceph node-exporter.c for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:51:52.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T22:51:53.140 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:53.140 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi116", "caps": []}]: dispatch 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]: dispatch 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: Reconfiguring daemon agent.smithi052 on smithi052 2024-03-11T22:51:53.141 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:53 smithi116 ceph-mon[27552]: Reconfiguring daemon agent.smithi116 on smithi116 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi116", "caps": []}]: dispatch 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]: dispatch 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: Reconfiguring daemon agent.smithi052 on smithi052 2024-03-11T22:51:53.321 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:53 smithi052 ceph-mon[25824]: Reconfiguring daemon agent.smithi116 on smithi116 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi052", "caps": []}]: dispatch 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi116", "caps": []}]: dispatch 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi152", "caps": []}]: dispatch 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: Reconfiguring daemon agent.smithi052 on smithi052 2024-03-11T22:51:53.329 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:53 smithi152 ceph-mon[27791]: Reconfiguring daemon agent.smithi116 on smithi116 2024-03-11T22:51:53.684 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:54.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:54 smithi052 ceph-mon[25824]: Reconfiguring daemon agent.smithi152 on smithi152 2024-03-11T22:51:54.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:54 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:54.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:54 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:54.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:54 smithi152 ceph-mon[27791]: Reconfiguring daemon agent.smithi152 on smithi152 2024-03-11T22:51:54.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:54 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:54.830 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:54 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:54.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:54 smithi116 ceph-mon[27552]: Reconfiguring daemon agent.smithi152 on smithi152 2024-03-11T22:51:54.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:54 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:54.988 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:54 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:55.645 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: agent 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: agent 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout: host_pattern: '*' 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:50:56.149017Z' 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout: running: 0 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-11T22:50:57.662484Z service:agent [INFO] "service was created" 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.646 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: alertmanager 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: alertmanager 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=a 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:50:30.512272Z' 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: - 9093 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: - 9094 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: running: 0 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-11T22:51:17.934094Z service:alertmanager [INFO] "service was created" 2024-03-11T22:51:55.647 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: grafana 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: grafana 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi116=a 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout:spec: 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: anonymous_access: true 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: protocol: https 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:50:33.315548Z' 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: - 3000 2024-03-11T22:51:55.648 INFO:teuthology.orchestra.run.smithi052.stdout: running: 0 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-11T22:51:33.972699Z service:grafana [INFO] "service was created" 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: mgr 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: mgr 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout: count: 3 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=y 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi116=x 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi152=z 2024-03-11T22:51:55.649 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:48:47.944704Z' 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-11T22:50:35.515575Z' 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: mon 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: mon 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: count: 3 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052:172.21.15.52=a 2024-03-11T22:51:55.650 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi116:172.21.15.116=b 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi152:172.21.15.152=c 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:48:20.468047Z' 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-11T22:50:35.515493Z' 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: node-exporter 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: node-exporter 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: count: 3 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=a 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi116=b 2024-03-11T22:51:55.651 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi152=c 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:50:27.640859Z' 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: - 9100 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: running: 0 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-11T22:51:52.061095Z service:node-exporter [INFO] "service was created" 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: osd 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: osd 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:unmanaged: true 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:spec: 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: filter_logic: AND 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: objectstore: bluestore 2024-03-11T22:51:55.652 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: container_image_id: a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-11T22:50:35.515653Z' 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: running: 3 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: size: 3 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.653 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: prometheus 2024-03-11T22:51:55.654 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: prometheus 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi116=a 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:50:24.877662Z' 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: - 9095 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: running: 0 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout:events: 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout:- 2024-03-11T22:51:52.068798Z service:prometheus [INFO] "service was created" 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout:--- 2024-03-11T22:51:55.655 INFO:teuthology.orchestra.run.smithi052.stdout:service_type: rgw 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout:service_id: foo.a 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout:service_name: rgw.foo.a 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout:placement: 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: count: 1 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: hosts: 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: - smithi052=foo.a 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout:status: 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: created: '2024-03-11T22:50:25.261281Z' 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: last_refresh: '2024-03-11T22:50:48.384596Z' 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: ports: 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: - 80 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: running: 1 2024-03-11T22:51:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: size: 1 2024-03-11T22:51:55.847 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:55 smithi116 ceph-mon[27552]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:55.847 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:55 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:55.955 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:55 smithi152 ceph-mon[27791]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:55.955 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:55 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:56.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:55 smithi052 ceph-mon[25824]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:56.042 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:55 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:56.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:56 smithi116 ceph-mon[27552]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:51:56.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:56.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:56 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:57.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:56 smithi152 ceph-mon[27791]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:51:57.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:57.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:56 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:57.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:56 smithi052 ceph-mon[25824]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:51:57.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:51:57.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:56 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:51:57.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 09234c40-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T22:51:57.812 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:57 smithi052 ceph-mon[25824]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:57.835 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:57 smithi116 ceph-mon[27552]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:57.923 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:57 smithi152 ceph-mon[27791]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:58.234 INFO:teuthology.orchestra.run.smithi052.stderr:Inferring config /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:51:59.829 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:51:59 smithi152 ceph-mon[27791]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:51:59.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:51:59 smithi116 ceph-mon[27552]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:00.071 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:51:59 smithi052 ceph-mon[25824]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:01.732 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:01 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: ::ffff:172.21.15.116 - - [11/Mar/2024:22:52:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:52:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:01 smithi116 ceph-mon[27552]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:01 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:01.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:01 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:02.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:01 smithi052 ceph-mon[25824]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:02.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:01 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:02.008 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:01 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:02.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:01 smithi152 ceph-mon[27791]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:02.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:01 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:02.079 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:01 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:02.611 INFO:teuthology.orchestra.run.smithi052.stdout:osd 3 85s ago - 2024-03-11T22:52:03.570 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:03 smithi052 ceph-mon[25824]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:03.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:03 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:03.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:03 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:03.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:03 smithi052 ceph-mon[25824]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:52:03.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:03 smithi152 ceph-mon[27791]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:03.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:03 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:03.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:03 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:03.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:03 smithi152 ceph-mon[27791]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:52:03.584 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:03 smithi116 ceph-mon[27552]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:03.584 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:03 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:03.584 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:03 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:03.584 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:03 smithi116 ceph-mon[27552]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:52:04.819 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T22:52:04.829 INFO:tasks.cephadm:Teardown begin 2024-03-11T22:52:04.830 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:52:04.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:52:04.898 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:52:04.933 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T22:52:04.933 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:52:04.948 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:52:04.965 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:52:04.989 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T22:52:04.989 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T22:52:04.989 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:52:05.070 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 systemd[1]: Stopping Ceph mon.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:05.566 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-mon[25824]: pgmap v37: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:05.571 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:05.572 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-mon[25824]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:05.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:05 smithi152 ceph-mon[27791]: pgmap v37: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:05.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:05 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:05.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:05 smithi152 ceph-mon[27791]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:05.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:05 smithi116 ceph-mon[27552]: pgmap v37: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:52:05.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:05 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:05.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:05 smithi116 ceph-mon[27552]: from='mgr.14442 172.21.15.52:0/3377310793' entity='mgr.y' 2024-03-11T22:52:05.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a[25801]: 2024-03-11T22:52:05.560+0000 7fb395f8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:05.820 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a[25801]: 2024-03-11T22:52:05.560+0000 7fb395f8f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:05 smithi052 podman[53605]: 2024-03-11 22:52:05.724068694 +0000 UTC m=+0.446168517 container died d0af2840eadd0c8cde49aa1e74191719a7ce460dc5f180c9dcfdb08125c8bdfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True) 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:05] ENGINE Bus STOPPING 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:05] ENGINE Bus STOPPED 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:05] ENGINE Bus STARTING 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:05] ENGINE Serving on http://:::9283 2024-03-11T22:52:05.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:05 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:05] ENGINE Bus STARTED 2024-03-11T22:52:06.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:06 smithi052 podman[53605]: 2024-03-11 22:52:06.208053483 +0000 UTC m=+0.930153301 container cleanup d0af2840eadd0c8cde49aa1e74191719a7ce460dc5f180c9dcfdb08125c8bdfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:52:06.320 INFO:journalctl@ceph.mon.a.smithi052.stdout:Mar 11 22:52:06 smithi052 bash[53605]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:52:06.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T22:52:06.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:06.683 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T22:52:06.684 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T22:52:06.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.b 2024-03-11T22:52:06.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:06 smithi116 systemd[1]: Stopping Ceph mon.b for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:08.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:08 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-b[27529]: 2024-03-11T22:52:08.331+0000 7feb571ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:08.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:08 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-b[27529]: 2024-03-11T22:52:08.331+0000 7feb571ee700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:52:08.737 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:08 smithi116 podman[40842]: 2024-03-11 22:52:08.525819815 +0000 UTC m=+0.317146496 container died 8a1c8d1a55fdb5a1cd919ecc3bd3d3ef0615532b8e5379fa4d1e0a71e36bc469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T22:52:08.820 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:08] ENGINE Bus STOPPING 2024-03-11T22:52:08.820 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:52:08.820 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:08] ENGINE Bus STOPPED 2024-03-11T22:52:08.820 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:08] ENGINE Bus STARTING 2024-03-11T22:52:08.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:08] ENGINE Serving on http://:::9283 2024-03-11T22:52:08.821 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:08 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: [11/Mar/2024:22:52:08] ENGINE Bus STARTED 2024-03-11T22:52:09.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:08 smithi116 podman[40842]: 2024-03-11 22:52:08.828593347 +0000 UTC m=+0.619920035 container cleanup 8a1c8d1a55fdb5a1cd919ecc3bd3d3ef0615532b8e5379fa4d1e0a71e36bc469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-11T22:52:09.034 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 11 22:52:08 smithi116 bash[40842]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-b 2024-03-11T22:52:09.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T22:52:09.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:09.292 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T22:52:09.292 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T22:52:09.292 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.c 2024-03-11T22:52:09.579 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:09 smithi152 systemd[1]: Stopping Ceph mon.c for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:09.885 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:09 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c[27755]: 2024-03-11T22:52:09.622+0000 7ffb37799700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:09.886 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:09 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c[27755]: 2024-03-11T22:52:09.622+0000 7ffb37799700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:52:09.886 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:09 smithi152 podman[45543]: 2024-03-11 22:52:09.704874733 +0000 UTC m=+0.221444897 container died 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:52:09.886 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:09 smithi152 podman[45543]: 2024-03-11 22:52:09.881166203 +0000 UTC m=+0.397736412 container exec_died 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:52:10.138 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:10 smithi152 podman[45543]: 2024-03-11 22:52:10.111849265 +0000 UTC m=+0.628419438 container cleanup 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True) 2024-03-11T22:52:10.139 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:10 smithi152 bash[45543]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c 2024-03-11T22:52:10.689 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T22:52:10.713 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:10 smithi152 podman[45562]: 2024-03-11 22:52:10.438969146 +0000 UTC m=+0.731674470 container remove 074b8af07f2c84d3edaae2eacf3405ec02395c14b9a7a85db478eefbc19b51e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:52:10.713 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:10 smithi152 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-11T22:52:10.713 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:10 smithi152 systemd[1]: Stopped Ceph mon.c for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:52:10.713 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 11 22:52:10 smithi152 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@mon.c.service: Consumed 4.132s CPU time. 2024-03-11T22:52:10.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:10.738 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T22:52:10.738 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T22:52:10.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:52:11.053 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:10 smithi052 systemd[1]: Stopping Ceph mgr.y for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:11.320 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:11 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:52:11.047+0000 7f0e04652700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:11.321 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:11 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y[26077]: 2024-03-11T22:52:11.047+0000 7f0e04652700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T22:52:11.321 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:11 smithi052 podman[53878]: 2024-03-11 22:52:11.079176048 +0000 UTC m=+0.150857669 container died afc4e2cd73382a16ce27c701dfb21488e08b4d4c393f15c77f6bc17f36200c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T22:52:11.726 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:11 smithi052 podman[53878]: 2024-03-11 22:52:11.487120968 +0000 UTC m=+0.558802589 container cleanup afc4e2cd73382a16ce27c701dfb21488e08b4d4c393f15c77f6bc17f36200c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T22:52:11.726 INFO:journalctl@ceph.mgr.y.smithi052.stdout:Mar 11 22:52:11 smithi052 bash[53878]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-y 2024-03-11T22:52:11.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T22:52:11.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:11.888 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T22:52:11.888 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T22:52:11.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.x 2024-03-11T22:52:12.227 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:52:11 smithi116 systemd[1]: Stopping Ceph mgr.x for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:12.487 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:52:12 smithi116 podman[40973]: 2024-03-11 22:52:12.222870577 +0000 UTC m=+0.145177748 container died 72598752c6e66dc8d7a89216f38bf12deb920d46aaf09ce33b56fab126842542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T22:52:12.871 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:52:12 smithi116 podman[40973]: 2024-03-11 22:52:12.573905028 +0000 UTC m=+0.496212207 container cleanup 72598752c6e66dc8d7a89216f38bf12deb920d46aaf09ce33b56fab126842542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:52:12.871 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 11 22:52:12 smithi116 bash[40973]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-x 2024-03-11T22:52:13.030 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T22:52:13.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:13.066 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T22:52:13.066 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T22:52:13.066 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.z 2024-03-11T22:52:13.386 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:52:13 smithi152 systemd[1]: Stopping Ceph mgr.z for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:13.663 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:52:13 smithi152 podman[45675]: 2024-03-11 22:52:13.381879264 +0000 UTC m=+0.129126809 container died 6265e2d4f5ac3e7c74b282ea703c473d45d1818faa81a3639c494f58d90c36c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T22:52:14.005 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:52:13 smithi152 podman[45675]: 2024-03-11 22:52:13.74968655 +0000 UTC m=+0.496934099 container cleanup 6265e2d4f5ac3e7c74b282ea703c473d45d1818faa81a3639c494f58d90c36c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:52:14.005 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 11 22:52:13 smithi152 bash[45675]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-mgr-z 2024-03-11T22:52:14.171 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T22:52:14.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:14.206 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T22:52:14.206 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T22:52:14.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.0 2024-03-11T22:52:14.532 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:14 smithi052 systemd[1]: Stopping Ceph osd.0 for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:14.820 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:14 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0[38157]: 2024-03-11T22:52:14.526+0000 7f27cd18d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:14.820 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:14 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0[38157]: 2024-03-11T22:52:14.526+0000 7f27cd18d700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-11T22:52:14.821 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:14 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0[38157]: 2024-03-11T22:52:14.526+0000 7f27cd18d700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:52:20.320 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:20 smithi052 podman[54020]: 2024-03-11 22:52:20.058903753 +0000 UTC m=+5.663693648 container died be36f578cff22bae38e11710a7d6179f66728bdf5c79d284309d5db7ec7b95dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:52:20.637 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:20 smithi052 podman[54020]: 2024-03-11 22:52:20.448129371 +0000 UTC m=+6.052919272 container cleanup be36f578cff22bae38e11710a7d6179f66728bdf5c79d284309d5db7ec7b95dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T22:52:20.637 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:20 smithi052 bash[54020]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:52:21.070 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:20 smithi052 podman[54679]: 2024-03-11 22:52:20.632570936 +0000 UTC m=+0.648575612 container remove be36f578cff22bae38e11710a7d6179f66728bdf5c79d284309d5db7ec7b95dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T22:52:21.334 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:21 smithi052 podman[54802]: 2024-03-11 22:52:21.194918598 +0000 UTC m=+0.017749437 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:52:21.334 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:21 smithi052 podman[54802]: 2024-03-11 22:52:21.292897079 +0000 UTC m=+0.115727897 container create 9dcb4c11406e9b8441b598912447fdc8ee10abe7f66c20b38d273355d023d379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T22:52:21.633 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:21 smithi052 podman[54802]: 2024-03-11 22:52:21.529294521 +0000 UTC m=+0.352125347 container init 9dcb4c11406e9b8441b598912447fdc8ee10abe7f66c20b38d273355d023d379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T22:52:21.634 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:21 smithi052 podman[54802]: 2024-03-11 22:52:21.534999478 +0000 UTC m=+0.357830302 container start 9dcb4c11406e9b8441b598912447fdc8ee10abe7f66c20b38d273355d023d379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:52:21.634 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:21 smithi052 podman[54802]: 2024-03-11 22:52:21.629074421 +0000 UTC m=+0.451905248 container attach 9dcb4c11406e9b8441b598912447fdc8ee10abe7f66c20b38d273355d023d379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T22:52:22.070 INFO:journalctl@ceph.osd.0.smithi052.stdout:Mar 11 22:52:21 smithi052 podman[54802]: 2024-03-11 22:52:21.830272226 +0000 UTC m=+0.653103045 container died 9dcb4c11406e9b8441b598912447fdc8ee10abe7f66c20b38d273355d023d379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:52:22.466 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T22:52:22.507 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:22.507 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T22:52:22.507 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T22:52:22.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.1 2024-03-11T22:52:22.826 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:22 smithi116 systemd[1]: Stopping Ceph osd.1 for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:23.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:22 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1[33089]: 2024-03-11T22:52:22.820+0000 7fa5b8737700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:23.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:22 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1[33089]: 2024-03-11T22:52:22.820+0000 7fa5b8737700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-11T22:52:23.237 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:22 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1[33089]: 2024-03-11T22:52:22.820+0000 7fa5b8737700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:52:28.540 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:28 smithi116 podman[41824]: 2024-03-11 22:52:28.259898866 +0000 UTC m=+5.554590398 container died 45ce9126aa80946f2293af886408a89ad7ce7b20a447decb4a2422efa3579423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:52:28.831 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:28 smithi116 podman[41824]: 2024-03-11 22:52:28.634310727 +0000 UTC m=+5.929002260 container cleanup 45ce9126aa80946f2293af886408a89ad7ce7b20a447decb4a2422efa3579423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T22:52:28.831 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:28 smithi116 bash[41824]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:52:29.236 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:28 smithi116 podman[41857]: 2024-03-11 22:52:28.826886847 +0000 UTC m=+0.560741895 container remove 45ce9126aa80946f2293af886408a89ad7ce7b20a447decb4a2422efa3579423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T22:52:29.551 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:29 smithi116 podman[41954]: 2024-03-11 22:52:29.146573905 +0000 UTC m=+0.018883662 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:52:29.551 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:29 smithi116 podman[41954]: 2024-03-11 22:52:29.253056672 +0000 UTC m=+0.125366415 container create ed9b8ea687518170db916076debab131be60aa511dcb4fd3f95058c24c536bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T22:52:29.551 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:29 smithi116 podman[41954]: 2024-03-11 22:52:29.455294042 +0000 UTC m=+0.327603785 container init ed9b8ea687518170db916076debab131be60aa511dcb4fd3f95058c24c536bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T22:52:29.551 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:29 smithi116 podman[41954]: 2024-03-11 22:52:29.460436597 +0000 UTC m=+0.332746341 container start ed9b8ea687518170db916076debab131be60aa511dcb4fd3f95058c24c536bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T22:52:29.551 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:29 smithi116 podman[41954]: 2024-03-11 22:52:29.546667487 +0000 UTC m=+0.418977233 container attach ed9b8ea687518170db916076debab131be60aa511dcb4fd3f95058c24c536bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T22:52:29.987 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 11 22:52:29 smithi116 podman[41954]: 2024-03-11 22:52:29.752895914 +0000 UTC m=+0.625205651 container died ed9b8ea687518170db916076debab131be60aa511dcb4fd3f95058c24c536bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:52:30.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T22:52:30.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:30.412 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T22:52:30.412 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T22:52:30.412 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.2 2024-03-11T22:52:30.722 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:30 smithi152 systemd[1]: Stopping Ceph osd.2 for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:31.078 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:30 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2[34390]: 2024-03-11T22:52:30.716+0000 7fe84e191700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:31.079 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:30 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2[34390]: 2024-03-11T22:52:30.716+0000 7fe84e191700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-11T22:52:31.079 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:30 smithi152 ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2[34390]: 2024-03-11T22:52:30.716+0000 7fe84e191700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:52:36.319 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:36 smithi152 podman[46377]: 2024-03-11 22:52:36.051346973 +0000 UTC m=+5.445978099 container died 7c6f38f4bcbc7ec5a5ce8bfead0267513ca1f8ba30fd850fa0ef5040d6e41fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:52:36.578 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:36 smithi152 podman[46377]: 2024-03-11 22:52:36.388701103 +0000 UTC m=+5.783332229 container cleanup 7c6f38f4bcbc7ec5a5ce8bfead0267513ca1f8ba30fd850fa0ef5040d6e41fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T22:52:36.579 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:36 smithi152 bash[46377]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:52:36.922 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:36 smithi152 podman[46400]: 2024-03-11 22:52:36.808488242 +0000 UTC m=+0.752330270 container remove 7c6f38f4bcbc7ec5a5ce8bfead0267513ca1f8ba30fd850fa0ef5040d6e41fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T22:52:38.621 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:38 smithi152 podman[46711]: 2024-03-11 22:52:38.366581812 +0000 UTC m=+0.108126593 container create f038fb15187cbe18062850104cef087ae6b5e7dde089c08f206d33a07844d56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:52:38.621 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:38 smithi152 podman[46711]: 2024-03-11 22:52:38.276168789 +0000 UTC m=+0.017713578 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:52:38.622 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:38 smithi152 podman[46711]: 2024-03-11 22:52:38.610865474 +0000 UTC m=+0.352410274 container init f038fb15187cbe18062850104cef087ae6b5e7dde089c08f206d33a07844d56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:52:38.925 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:38 smithi152 podman[46711]: 2024-03-11 22:52:38.616567132 +0000 UTC m=+0.358112024 container start f038fb15187cbe18062850104cef087ae6b5e7dde089c08f206d33a07844d56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T22:52:38.925 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:38 smithi152 podman[46711]: 2024-03-11 22:52:38.719566817 +0000 UTC m=+0.461111610 container attach f038fb15187cbe18062850104cef087ae6b5e7dde089c08f206d33a07844d56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-11T22:52:39.219 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 11 22:52:38 smithi152 podman[46711]: 2024-03-11 22:52:38.921120273 +0000 UTC m=+0.662665047 container died f038fb15187cbe18062850104cef087ae6b5e7dde089c08f206d33a07844d56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:52:39.490 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T22:52:39.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:39.532 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T22:52:39.532 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T22:52:39.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T22:52:39.820 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:39 smithi052 systemd[1]: Stopping Ceph rgw.foo.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:40.320 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:39 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a[41188]: 2024-03-11T22:52:39.865+0000 7f9f3a56f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:52:40.320 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:39 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a[41188]: 2024-03-11T22:52:39.865+0000 7f9f48ddfac0 -1 shutting down 2024-03-11T22:52:47.987 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:47.725Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-03-11T22:52:47.987 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:47 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:47.725Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.52:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.52:8765: connect: connection refused" 2024-03-11T22:52:50.218 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:49 smithi052 bash[55597]: time="2024-03-11T22:52:49Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T22:52:50.218 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:49 smithi052 podman[55597]: 2024-03-11 22:52:49.886866648 +0000 UTC m=+10.148279780 container died d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:52:50.570 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 podman[55597]: 2024-03-11 22:52:50.404263477 +0000 UTC m=+10.665676660 container cleanup d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:52:50.570 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 bash[55597]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T22:52:50.857 DEBUG:teuthology.orchestra.run.smithi052:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T22:52:50.881 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 podman[55622]: 2024-03-11 22:52:50.6221321 +0000 UTC m=+0.730265131 container remove d5b67d4081225b23cc3b5f7449ed61d0c6f8372ecbf1700eada778f976a2a7c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T22:52:50.881 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T22:52:50.881 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-11T22:52:50.881 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 systemd[1]: Stopped Ceph rgw.foo.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:52:50.881 INFO:journalctl@ceph.rgw.foo.a.smithi052.stdout:Mar 11 22:52:50 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service: Consumed 1.670s CPU time. 2024-03-11T22:52:50.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:50.895 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T22:52:50.895 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T22:52:50.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-09234c40-dff9-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T22:52:51.186 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:50 smithi116 systemd[1]: Stopping Ceph prometheus.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:51.487 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:52:51.487 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:52:51.487 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:52:51.487 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:52:51.487 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a[35924]: ts=2024-03-11T22:52:51.182Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:52:51.488 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 podman[42840]: 2024-03-11 22:52:51.193054558 +0000 UTC m=+0.113499461 container died be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:51.816 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 podman[42840]: 2024-03-11 22:52:51.487212202 +0000 UTC m=+0.407657104 container cleanup be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:51.816 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 bash[42840]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:52:51.817 INFO:journalctl@ceph.prometheus.a.smithi116.stdout:Mar 11 22:52:51 smithi116 podman[42860]: 2024-03-11 22:52:51.705106847 +0000 UTC m=+0.515908252 container remove be570ca20cb2df3b251d0e799efd891e14bf96d370e97ef34c512d9a5007c244 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:51.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-09234c40-dff9-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T22:52:51.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:52:51.872 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T22:52:51.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:52:52.407 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:52:54.314 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:54 smithi052 systemd[1]: Stopping Ceph alertmanager.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:54.570 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:54 smithi052 ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a[48161]: ts=2024-03-11T22:52:54.309Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:52:54.570 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:54 smithi052 podman[56185]: 2024-03-11 22:52:54.340224161 +0000 UTC m=+0.140026364 container died 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:54.953 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:54 smithi052 podman[56185]: 2024-03-11 22:52:54.765011802 +0000 UTC m=+0.564814075 container cleanup 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:54.954 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:54 smithi052 bash[56185]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:52:55.227 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:54 smithi052 podman[56204]: 2024-03-11 22:52:54.949414947 +0000 UTC m=+0.611410276 container remove 01d0d8b20aca11d0b804b680acaa3f160dc026f8ae6c2ec9e5cd5e9899e96cec (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:55.228 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:55 smithi052 podman[56204]: 2024-03-11 22:52:55.016595571 +0000 UTC m=+0.678590901 volume remove 026ecc2079ee9ad888caffa275da05bd75d2678b3363f19988786d3cdbd62620 2024-03-11T22:52:55.570 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:55 smithi052 systemd[1]: Stopping Ceph node-exporter.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:55.570 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:55 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:52:55.570 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:55 smithi052 systemd[1]: Stopped Ceph alertmanager.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:52:55.570 INFO:journalctl@ceph.alertmanager.a.smithi052.stdout:Mar 11 22:52:55 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.154s CPU time. 2024-03-11T22:52:56.008 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:55 smithi052 podman[56322]: 2024-03-11 22:52:55.73324125 +0000 UTC m=+0.128327405 container died 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:56.320 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 podman[56322]: 2024-03-11 22:52:56.185986944 +0000 UTC m=+0.581073098 container cleanup 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:56.320 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 bash[56322]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T22:52:56.802 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 podman[56342]: 2024-03-11 22:52:56.445454425 +0000 UTC m=+0.717318485 container remove 74e2ec64e6cb4ef9ac2572e8c0e72dd3d406986039a92e3891f85b3a8d6dc828 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:52:56.802 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:52:56.802 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T22:52:56.802 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 systemd[1]: Stopped Ceph node-exporter.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:52:56.803 INFO:journalctl@ceph.node-exporter.a.smithi052.stdout:Mar 11 22:52:56 smithi052 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.911s CPU time. 2024-03-11T22:52:57.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:52:57.386 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:52:58.688 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:58 smithi116 systemd[1]: Stopping Ceph grafana.a for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:58.987 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:58 smithi116 podman[43414]: 2024-03-11 22:52:58.683386392 +0000 UTC m=+0.108320511 container died 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, version=8.5, release=239.1651231664, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T22:52:59.408 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 podman[43414]: 2024-03-11 22:52:59.002543551 +0000 UTC m=+0.427477660 container cleanup 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, version=8.5, io.openshift.expose-services=, release=239.1651231664, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-11T22:52:59.408 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 bash[43414]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a 2024-03-11T22:52:59.408 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 podman[43434]: 2024-03-11 22:52:59.220211491 +0000 UTC m=+0.540098986 container remove 4406a0b92927d1f17fd0b4d0dc03f4559b7f676564d4e9ed7f4ec71135bf66d6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-grafana-a, maintainer=Paul Cuzner , name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, distribution-scope=public, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, version=8.5, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T22:52:59.408 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:52:59.736 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:52:59 smithi116 systemd[1]: Stopping Ceph node-exporter.b for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:52:59.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T22:52:59.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 systemd[1]: Stopped Ceph grafana.a for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:52:59.737 INFO:journalctl@ceph.grafana.a.smithi116.stdout:Mar 11 22:52:59 smithi116 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@grafana.a.service: Consumed 4.005s CPU time. 2024-03-11T22:53:00.161 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:52:59 smithi116 podman[43550]: 2024-03-11 22:52:59.870208558 +0000 UTC m=+0.116869954 container died 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:53:00.487 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 podman[43550]: 2024-03-11 22:53:00.256231057 +0000 UTC m=+0.502892453 container cleanup 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:53:00.487 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 bash[43550]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T22:53:00.893 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 podman[43569]: 2024-03-11 22:53:00.490445445 +0000 UTC m=+0.623624722 container remove 4176bb36a5f4bc5a5833112bab0ca553fba1b76c30b75c7cc4a1504b7f2edb51 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:53:00.894 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:53:00.894 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T22:53:00.894 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 systemd[1]: Stopped Ceph node-exporter.b for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:00.894 INFO:journalctl@ceph.node-exporter.b.smithi116.stdout:Mar 11 22:53:00 smithi116 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 1.964s CPU time. 2024-03-11T22:53:01.252 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:53:01.447 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:02.579 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:02 smithi152 systemd[1]: Stopping Ceph node-exporter.c for 09234c40-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:53:02.579 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:02 smithi152 podman[47606]: 2024-03-11 22:53:02.533811894 +0000 UTC m=+0.106674664 container died ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:53:03.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:02 smithi152 podman[47606]: 2024-03-11 22:53:02.903519271 +0000 UTC m=+0.476382037 container cleanup ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:53:03.079 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:02 smithi152 bash[47606]: ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c 2024-03-11T22:53:03.485 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:03 smithi152 podman[47625]: 2024-03-11 22:53:03.171264256 +0000 UTC m=+0.642345921 container remove ae78b376d19788af65c5dc4fb7a69bc8bd7b68dfae1acb1c814fdd510cc70cb4 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-09234c40-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:53:03.485 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:03 smithi152 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:53:03.485 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:03 smithi152 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-11T22:53:03.485 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:03 smithi152 systemd[1]: Stopped Ceph node-exporter.c for 09234c40-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:03.486 INFO:journalctl@ceph.node-exporter.c.smithi152.stdout:Mar 11 22:53:03 smithi152 systemd[1]: ceph-09234c40-dff9-11ee-95c8-87774f69a715@node-exporter.c.service: Consumed 1.922s CPU time. 2024-03-11T22:53:03.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:04.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:04.051 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:04.080 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T22:53:04.082 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi052/crash 2024-03-11T22:53:04.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:53:04.147 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:53:04.147 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:53:04.150 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi116/crash 2024-03-11T22:53:04.151 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:53:04.201 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:53:04.201 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:53:04.203 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi152/crash 2024-03-11T22:53:04.204 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:53:04.245 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/09234c40-dff9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:53:04.245 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:53:04.245 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T22:53:04.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:53:04.277 INFO:teuthology.orchestra.run.smithi052.stdout:2024-03-11T22:48:39.720185+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:53:04.278 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T22:53:04.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[SEC\]' /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:53:04.349 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]' /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:53:04.419 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[WRN\]' /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:53:04.489 INFO:teuthology.orchestra.run.smithi052.stdout:2024-03-11T22:48:39.720185+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:53:04.490 INFO:tasks.cephadm:Compressing logs... 2024-03-11T22:53:04.490 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:53:04.532 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:53:04.534 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:53:04.558 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:53:04.559 INFO:teuthology.orchestra.run.smithi052.stderr:: No such file or directory 2024-03-11T22:53:04.559 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T22:53:04.559 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:53:04.559 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T22:53:04.559 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T22:53:04.561 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T22:53:04.561 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.y.log: 57.6% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T22:53:04.561 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:53:04.561 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:53:04.561 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:53:04.561 INFO:teuthology.orchestra.run.smithi052.stderr: 89.7%/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log: -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 82.2% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:53:04.562 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-11T22:53:04.562 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log: 85.2% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:53:04.562 INFO:teuthology.orchestra.run.smithi052.stderr: 84.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:53:04.566 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:53:04.566 INFO:teuthology.orchestra.run.smithi152.stderr:: No such file or directory 2024-03-11T22:53:04.566 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:53:04.566 INFO:teuthology.orchestra.run.smithi152.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr: 80.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log: 80.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr: 89.8% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr: 84.9% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T22:53:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:: No such file or directory 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 80.2% 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.b.log: 84.8% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:53:04.568 INFO:teuthology.orchestra.run.smithi116.stderr: 77.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:53:04.569 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T22:53:04.577 INFO:teuthology.orchestra.run.smithi052.stderr: 90.2% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:53:04.580 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:53:04.588 INFO:teuthology.orchestra.run.smithi152.stderr: 90.2% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:53:04.623 INFO:teuthology.orchestra.run.smithi116.stderr: 92.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T22:53:04.628 INFO:teuthology.orchestra.run.smithi152.stderr: 92.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T22:53:04.640 INFO:teuthology.orchestra.run.smithi052.stderr: 88.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T22:53:04.769 INFO:teuthology.orchestra.run.smithi052.stderr: 91.5% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T22:53:04.842 INFO:teuthology.orchestra.run.smithi052.stderr: 94.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:53:04.844 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-11T22:53:04.844 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m0.299s 2024-03-11T22:53:04.844 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m0.587s 2024-03-11T22:53:04.844 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.042s 2024-03-11T22:53:04.847 INFO:teuthology.orchestra.run.smithi152.stderr: 94.3% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T22:53:04.849 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-03-11T22:53:04.849 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m0.301s 2024-03-11T22:53:04.849 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m0.376s 2024-03-11T22:53:04.849 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.038s 2024-03-11T22:53:04.880 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/09234c40-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:53:04.881 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-11T22:53:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m0.334s 2024-03-11T22:53:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m0.395s 2024-03-11T22:53:04.882 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.043s 2024-03-11T22:53:04.882 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:53:04.882 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi052/log 2024-03-11T22:53:04.883 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:53:05.011 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi116/log 2024-03-11T22:53:05.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:53:05.109 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi152/log 2024-03-11T22:53:05.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:53:05.201 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:53:05.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T22:53:05.383 INFO:teuthology.orchestra.run.smithi052.stdout:Deleting cluster with fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:05.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T22:53:05.686 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:05.812 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 09234c40-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T22:53:05.995 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 09234c40-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:06.108 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:53:06.109 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:53:06.125 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:53:06.141 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:53:06.155 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:53:06.156 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T22:53:06.167 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_1... 2024-03-11T22:53:06.167 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_1 2024-03-11T22:53:06.310 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:53:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:06.311 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_2... 2024-03-11T22:53:06.312 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_2 2024-03-11T22:53:06.460 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:53:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:06.462 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_3... 2024-03-11T22:53:06.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_3 2024-03-11T22:53:06.630 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:53:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:06.631 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi052:/dev/vg_nvme/lv_4... 2024-03-11T22:53:06.631 DEBUG:teuthology.orchestra.run.smithi052:> sudo nvme disconnect -n lv_4 2024-03-11T22:53:06.794 INFO:teuthology.orchestra.run.smithi052.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:53:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:06.795 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-11T22:53:06.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/scratch_devs 2024-03-11T22:53:06.822 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi116:/dev/vg_nvme/lv_1... 2024-03-11T22:53:06.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo nvme disconnect -n lv_1 2024-03-11T22:53:06.969 INFO:teuthology.orchestra.run.smithi116.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:53:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:06.971 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi116:/dev/vg_nvme/lv_2... 2024-03-11T22:53:06.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo nvme disconnect -n lv_2 2024-03-11T22:53:07.115 INFO:teuthology.orchestra.run.smithi116.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:53:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:07.116 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi116:/dev/vg_nvme/lv_3... 2024-03-11T22:53:07.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo nvme disconnect -n lv_3 2024-03-11T22:53:07.289 INFO:teuthology.orchestra.run.smithi116.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:53:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:07.291 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi116:/dev/vg_nvme/lv_4... 2024-03-11T22:53:07.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo nvme disconnect -n lv_4 2024-03-11T22:53:07.457 INFO:teuthology.orchestra.run.smithi116.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:53:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:07.460 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-11T22:53:07.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/scratch_devs 2024-03-11T22:53:07.487 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi152:/dev/vg_nvme/lv_1... 2024-03-11T22:53:07.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo nvme disconnect -n lv_1 2024-03-11T22:53:07.641 INFO:teuthology.orchestra.run.smithi152.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:53:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:07.643 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi152:/dev/vg_nvme/lv_2... 2024-03-11T22:53:07.643 DEBUG:teuthology.orchestra.run.smithi152:> sudo nvme disconnect -n lv_2 2024-03-11T22:53:07.817 INFO:teuthology.orchestra.run.smithi152.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:53:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:07.819 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi152:/dev/vg_nvme/lv_3... 2024-03-11T22:53:07.819 DEBUG:teuthology.orchestra.run.smithi152:> sudo nvme disconnect -n lv_3 2024-03-11T22:53:08.014 INFO:teuthology.orchestra.run.smithi152.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:53:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:08.024 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi152:/dev/vg_nvme/lv_4... 2024-03-11T22:53:08.024 DEBUG:teuthology.orchestra.run.smithi152:> sudo nvme disconnect -n lv_4 2024-03-11T22:53:08.178 INFO:teuthology.orchestra.run.smithi152.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:53:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:08.179 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-11T22:53:08.179 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/scratch_devs 2024-03-11T22:53:08.205 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:53:08.216 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:53:08.216 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:53:08.219 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:53:08.221 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:53:08.232 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:53:08.234 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:53:08.236 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:53:08.236 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-11T22:53:08.236 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +4184us[+4194us] +/- 128ms 2024-03-11T22:53:08.236 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -1249us[-1240us] +/- 66ms 2024-03-11T22:53:08.236 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -893us[ -893us] +/- 75ms 2024-03-11T22:53:08.236 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:53:08.238 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:53:08.238 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-11T22:53:08.238 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +4178us[+4218us] +/- 128ms 2024-03-11T22:53:08.238 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -1235us[-1195us] +/- 66ms 2024-03-11T22:53:08.238 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -902us[ -902us] +/- 75ms 2024-03-11T22:53:08.238 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:53:08.259 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:53:08.263 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:53:08.263 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-11T22:53:08.263 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +4208us[+4208us] +/- 128ms 2024-03-11T22:53:08.263 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -1253us[-1266us] +/- 66ms 2024-03-11T22:53:08.263 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -890us[ -903us] +/- 75ms 2024-03-11T22:53:08.263 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:53:08.263 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:53:08.273 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:53:08.274 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:53:08.284 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:53:08.365 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:53:08.433 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:53:08.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:53:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:08.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:53:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:08.564 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:53:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:08.593 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:53:08.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:53:08.619 INFO:teuthology.task.internal:Duration was 734.584311 seconds 2024-03-11T22:53:08.619 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:53:08.635 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:53:08.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:53:08.638 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:53:08.640 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:53:08.681 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:53:08.681 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:53:08.682 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:53:09.114 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:53:09.114 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-03-11T22:53:09.115 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:53:09.135 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-11T22:53:09.135 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:53:09.166 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-11T22:53:09.166 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:53:09.192 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:53:09.192 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:53:09.195 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:53:09.209 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:53:09.261 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:53:09.261 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:53:09.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:53:09.278 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:53:09.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:53:09.477 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:53:09.477 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:53:09.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:53:09.531 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:53:09.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:53:09.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:53:09.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:53:09.574 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:53:09.594 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-03-11T22:53:09.597 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-11T22:53:09.624 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-11T22:53:09.651 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:53:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:09.673 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:53:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:09.687 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:53:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:53:09.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:53:09.721 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:53:09.722 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi052 2024-03-11T22:53:09.722 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:53:09.760 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi116 2024-03-11T22:53:09.761 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:53:09.797 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010/remote/smithi152 2024-03-11T22:53:09.797 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:53:09.833 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:53:09.833 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:53:09.835 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:53:09.837 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:53:09.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:53:09.897 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:53:09.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:53:09.912 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:53:09.913 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:53:09.915 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:53:09.917 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:53:09.926 INFO:teuthology.orchestra.run.smithi052.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:53 /home/ubuntu/cephtest 2024-03-11T22:53:09.929 INFO:teuthology.orchestra.run.smithi116.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:53 /home/ubuntu/cephtest 2024-03-11T22:53:09.937 INFO:teuthology.orchestra.run.smithi152.stdout: 265312 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:53 /home/ubuntu/cephtest 2024-03-11T22:53:09.938 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:53:09.950 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:53:09.983 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:53:10.013 DEBUG:teuthology.nuke:shortname: smithi052 2024-03-11T22:53:10.013 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:53:10.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:47.048741', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ6Gf+B0HuNRcMpFwu9v0p0gvhNIdQRSh8FzF8xoZdz6YdL2deDMWQ1xBaIdmxFbJLfIYaRxCj59ve+B6sdXIms='} 2024-03-11T22:53:10.041 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-11T22:53:10.041 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:53:10.063 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:47.052356', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBov3hdMqTycbJjxnqLPIQidB5x5HWk2MHnK2fQIe9oA68FYVPffWOxDs88jqk8+Ijd+XnhgsbvCq66DgGDsB8='} 2024-03-11T22:53:10.067 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-11T22:53:10.067 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:53:10.098 INFO:teuthology.orchestra.console.smithi052:Power off 2024-03-11T22:53:10.098 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:53:10.106 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:31:47.050671', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/0u5N2avfiEkrHReHX0Ot8NwHpQuoiWhyxvau7Xh9aFedNpa7PwldNySfI/O29k9ar/fMn6EcIlTz8FDJQK/A='} 2024-03-11T22:53:10.121 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-03-11T22:53:10.121 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:10.140 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-11T22:53:10.141 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:53:10.150 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-11T22:53:10.162 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-11T22:53:10.162 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:10.176 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-11T22:53:10.176 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:53:10.184 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-11T22:53:10.198 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-11T22:53:10.198 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:10.229 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-11T22:53:14.151 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:14.186 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:14.230 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:14.263 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-11T22:53:14.296 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-11T22:53:14.339 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-11T22:53:18.267 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:18.297 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:18.341 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:18.380 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-03-11T22:53:18.407 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-11T22:53:18.452 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-11T22:53:18.482 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-03-11T22:53:18.508 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-11T22:53:18.554 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-11T22:53:18.658 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 734.5843107700348 failure_reason: '"2024-03-11T22:48:39.720185+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:53:18.658 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:53:18.735 INFO:teuthology.run:FAIL