2024-03-11T21:45:44.921 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T21:45:44.921 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947 --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} --verbose -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/orig.config.yaml 2024-03-11T21:45:44.951 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:45:45.030 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591947' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2aPs54Jdwcrh3ud3AlUTKuZhEbLl+WksdmQQRZUmtOq5s+e63jHRhrCCoscBSwF3q3ymTyEK3SpVJDsJM86g0= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBmDhu7mQ5IWqHxzvPYXJmeiS61UydZAuhIrjYjsMNocCQG9SQu1V9ykcPbZ0mDV90xyGOpANXGLhq5d5D4d0RU= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqMMaV3SbeaoIprq5WD33/YqRg8D24pJPwyQPRRX8Nv4a9GUzsBskHdgO7RvvSVoa8RX2UcW61aQTQSiAs3xK8= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T21:45:45.030 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T21:45:45.031 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T21:45:45.031 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T21:45:45.032 INFO:teuthology.task.internal:Checking packages... 2024-03-11T21:45:45.056 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T21:45:45.056 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T21:45:45.056 INFO:teuthology.packaging:ref: None 2024-03-11T21:45:45.056 INFO:teuthology.packaging:tag: None 2024-03-11T21:45:45.056 INFO:teuthology.packaging:branch: squid 2024-03-11T21:45:45.056 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:45:45.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T21:45:45.199 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T21:45:45.201 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T21:45:45.208 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T21:45:45.208 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T21:45:45.231 INFO:teuthology.task.internal:Saving configuration 2024-03-11T21:45:45.243 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T21:45:45.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:45:45.297 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:40:27.191340', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2aPs54Jdwcrh3ud3AlUTKuZhEbLl+WksdmQQRZUmtOq5s+e63jHRhrCCoscBSwF3q3ymTyEK3SpVJDsJM86g0='} 2024-03-11T21:45:45.319 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:40:27.194908', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBmDhu7mQ5IWqHxzvPYXJmeiS61UydZAuhIrjYjsMNocCQG9SQu1V9ykcPbZ0mDV90xyGOpANXGLhq5d5D4d0RU='} 2024-03-11T21:45:45.343 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:40:27.193184', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqMMaV3SbeaoIprq5WD33/YqRg8D24pJPwyQPRRX8Nv4a9GUzsBskHdgO7RvvSVoa8RX2UcW61aQTQSiAs3xK8='} 2024-03-11T21:45:45.343 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T21:45:45.350 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T21:45:45.350 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T21:45:45.350 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-11T21:45:45.350 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T21:45:45.452 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f768e636f70>, signals=[15]) 2024-03-11T21:45:45.452 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T21:45:45.458 INFO:teuthology.task.internal:Opening connections... 2024-03-11T21:45:45.458 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-11T21:45:45.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:45:45.554 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-03-11T21:45:45.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:45:45.646 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-03-11T21:45:45.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:45:45.737 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T21:45:45.743 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-11T21:45:45.761 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-11T21:45:45.761 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:45:45.818 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:45:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:45:45.820 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-11T21:45:45.841 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-03-11T21:45:45.859 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-03-11T21:45:45.859 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-03-11T21:45:45.916 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-03-11T21:45:45.916 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-03-11T21:45:45.916 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:45:45.917 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:45:45.918 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-03-11T21:45:45.941 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-03-11T21:45:45.958 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-03-11T21:45:45.958 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-03-11T21:45:46.015 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-03-11T21:45:46.015 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-03-11T21:45:46.015 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-03-11T21:45:46.015 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:45:46.016 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:45:46.016 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-03-11T21:45:46.040 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T21:45:46.050 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T21:45:46.081 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T21:45:46.081 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:45:46.086 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:45:46.088 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:45:46.104 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T21:45:46.163 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T21:45:46.163 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:45:46.165 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:45:46.168 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:45:46.298 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T21:45:46.370 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T21:45:46.385 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T21:45:46.385 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T21:45:46.385 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:45:46.386 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:45:46.386 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:45:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.404 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-11T21:45:46.404 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-11T21:45:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.405 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-11T21:45:46.461 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:45:46.461 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-03-11T21:45:46.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-03-11T21:45:46.462 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:45:46.463 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-03-11T21:45:46.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-03-11T21:45:46.464 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:45:46.464 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-361.el9.x86_64 2024-03-11T21:45:46.464 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-03-11T21:45:48.515 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T21:45:48.660 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:45:48.759 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:45:49.932 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-11T21:45:50.253 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-11T21:45:50.368 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T21:45:51.014 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-11T21:45:51.476 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-03-11T21:45:51.758 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 7.8 MB/s | 8.0 MB 00:01 2024-03-11T21:45:55.069 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 19 MB 00:02 2024-03-11T21:45:55.392 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-03-11T21:45:56.998 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 19 MB 00:04 2024-03-11T21:45:59.220 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-11T21:45:59.468 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-11T21:46:00.629 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2024-03-11T21:46:01.127 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-11T21:46:02.262 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-11T21:46:02.348 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-11T21:46:03.774 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-03-11T21:46:06.924 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-03-11T21:46:07.370 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 5.7 kB/s | 1.7 kB 00:00 2024-03-11T21:46:07.516 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 21 MB 00:06 2024-03-11T21:46:09.569 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:46:09.569 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:46:09.691 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-11T21:46:09.706 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-11T21:46:09.706 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-11T21:46:09.706 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-11T21:46:09.706 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-11T21:46:09.706 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-03-11T21:46:09.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:46:09.708 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 100 M 2024-03-11T21:46:09.708 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-03-11T21:46:09.708 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-11T21:46:10.705 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 6.8 MB/s | 6.0 MB 00:00 2024-03-11T21:46:11.891 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-11T21:46:12.405 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 8.1 MB/s | 21 MB 00:02 2024-03-11T21:46:12.412 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-11T21:46:14.021 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 9.4 MB/s | 39 MB 00:04 2024-03-11T21:46:14.106 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:46:14.106 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:46:14.197 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:46:14.213 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:46:14.214 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:46:14.214 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:46:14.214 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-11T21:46:14.214 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-11T21:46:14.214 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-03-11T21:46:14.214 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:46:14.215 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 100 M 2024-03-11T21:46:14.215 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-03-11T21:46:14.215 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-11T21:46:14.561 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 20 MB/s | 6.0 MB 00:00 2024-03-11T21:46:14.719 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:46:14.719 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:46:14.814 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-11T21:46:14.823 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-11T21:46:14.823 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-11T21:46:14.823 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-11T21:46:14.823 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-11T21:46:14.823 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:46:14.823 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:46:14.824 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 100 M 2024-03-11T21:46:14.825 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 123 M 2024-03-11T21:46:14.825 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-11T21:46:15.047 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 7.7 MB/s | 33 MB 00:04 2024-03-11T21:46:15.047 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:46:15.047 INFO:teuthology.orchestra.run.smithi151.stdout:Total 19 MB/s | 100 MB 00:05 2024-03-11T21:46:15.059 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 26 MB/s | 21 MB 00:00 2024-03-11T21:46:15.635 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 29 MB/s | 39 MB 00:01 2024-03-11T21:46:15.928 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-11T21:46:16.185 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 21 MB/s | 33 MB 00:01 2024-03-11T21:46:16.185 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:46:16.185 INFO:teuthology.orchestra.run.smithi176.stdout:Total 51 MB/s | 100 MB 00:01 2024-03-11T21:46:16.306 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-11T21:46:16.307 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-11T21:46:16.330 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.2 MB/s | 6.0 MB 00:01 2024-03-11T21:46:17.067 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-11T21:46:17.425 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-11T21:46:17.425 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-11T21:46:17.642 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-11T21:46:17.643 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-11T21:46:18.779 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-11T21:46:18.779 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-11T21:46:20.625 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-11T21:46:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:46:21.448 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:46:21.693 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-11T21:46:22.390 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:46:22.568 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:46:24.269 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:46:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:46:25.352 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:46:25.526 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:46:29.214 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:46:29.360 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:46:30.255 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:46:30.437 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:46:34.120 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:46:35.193 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:46:36.087 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.0 MB/s | 21 MB 00:21 2024-03-11T21:46:53.702 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 915 kB/s | 33 MB 00:37 2024-03-11T21:47:01.443 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 865 kB/s | 39 MB 00:46 2024-03-11T21:47:01.443 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:47:01.443 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.1 MB/s | 100 MB 00:46 2024-03-11T21:47:02.329 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-11T21:47:02.678 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-11T21:47:02.678 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-11T21:47:03.396 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:03.396 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:47:03.396 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:47:03.482 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:03.864 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:03.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:47:03.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:47:03.864 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:47:04.231 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:04.231 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:47:04.231 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:47:04.285 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:47:04.286 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-11T21:47:04.299 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-11T21:47:04.299 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-11T21:47:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:04.732 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:04.732 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:47:04.732 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:47:04.732 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:47:04.765 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:47:04.919 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:04.920 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:47:04.920 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:47:04.920 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:47:04.920 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:47:04.920 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T21:47:04.920 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:47:04.921 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:47:05.041 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.041 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:47:05.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-03-11T21:47:05.082 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-03-11T21:47:05.082 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-03-11T21:47:05.082 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-03-11T21:47:05.083 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:47:05.084 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:47:05.085 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:47:05.085 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:47:05.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:47:05.159 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-11T21:47:05.635 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-11T21:47:05.677 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:47:05.816 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.816 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:47:05.816 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:47:05.816 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:47:05.816 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:47:05.816 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T21:47:05.816 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:47:05.816 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:47:05.937 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:05.938 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:47:05.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:47:05.972 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:47:05.973 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:47:05.974 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:47:05.974 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:47:05.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:47:06.566 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-03-11T21:47:07.020 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:47:07.074 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-11T21:47:07.076 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-11T21:47:07.091 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.aSaAJ0l6gn 2024-03-11T21:47:07.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aSaAJ0l6gn 2024-03-11T21:47:07.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.aSaAJ0l6gn 2024-03-11T21:47:07.204 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-11T21:47:07.301 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.aSaAJ0l6gn is 6KB 2024-03-11T21:47:07.349 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.aSaAJ0l6gn 2024-03-11T21:47:07.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:47:07.427 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:47:07.427 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:47:07.428 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:47:07.428 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:47:07.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:47:07.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-03-11T21:47:07.870 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:47:07.944 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:47:07.981 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-03-11T21:47:07.983 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-11T21:47:07.993 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:47:07.998 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.LKNA27xwAh 2024-03-11T21:47:07.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LKNA27xwAh 2024-03-11T21:47:08.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.LKNA27xwAh 2024-03-11T21:47:08.203 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.LKNA27xwAh is 6KB 2024-03-11T21:47:08.252 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.LKNA27xwAh 2024-03-11T21:47:08.266 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:47:08.331 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:47:08.331 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:47:08.331 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:47:08.331 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:47:08.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:47:08.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-03-11T21:47:10.790 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:47:10.972 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:47:15.734 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:47:15.960 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:20.749 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:37.585 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:47:37.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:47:37.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:47:38.476 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:47:38.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-11T21:47:38.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:47:49.903 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:49.904 INFO:teuthology.orchestra.run.smithi189.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:47:49.904 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:47:49.986 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:50.465 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:50.466 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:47:50.466 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:47:50.466 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:47:50.897 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-11T21:47:51.417 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:47:51.616 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:51.616 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:47:51.617 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:47:51.617 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:47:51.617 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:47:51.617 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T21:47:51.617 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:47:51.617 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:47:51.739 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:47:51.739 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:47:51.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:47:51.784 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:47:51.785 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:47:51.786 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-03-11T21:47:51.786 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:47:51.786 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:47:51.786 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:47:51.787 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:47:51.787 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:47:52.304 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-03-11T21:47:53.659 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:47:53.691 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-03-11T21:47:53.694 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-03-11T21:47:53.708 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.bIDIe2g9Yy 2024-03-11T21:47:53.709 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bIDIe2g9Yy 2024-03-11T21:47:53.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.bIDIe2g9Yy 2024-03-11T21:47:53.919 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.bIDIe2g9Yy is 6KB 2024-03-11T21:47:53.939 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.bIDIe2g9Yy 2024-03-11T21:47:53.953 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:47:54.017 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:47:54.018 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:47:54.018 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:47:54.018 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:47:54.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:47:54.187 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-03-11T21:47:56.066 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-11T21:47:56.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-11T21:48:05.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:48:05.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:48:05.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-11T21:48:05.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:48:08.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-11T21:48:20.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:48:20.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:48:24.194 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T21:48:24.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:48:24.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:48:38.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-11T21:48:42.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-11T21:48:51.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:48:51.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:48:53.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:48:53.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:48:53.941 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-11T21:48:54.380 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:48:54.380 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:48:54.381 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-11T21:48:54.395 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:48:54.395 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:48:54.395 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:48:54.395 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:48:54.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-11T21:48:55.396 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:48:55.396 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:48:55.482 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-03-11T21:48:55.522 DEBUG:teuthology.parallel:result is None 2024-03-11T21:49:05.835 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:49:06.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:49:06.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:49:17.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-11T21:49:17.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:49:18.255 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-11T21:49:18.655 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-11T21:49:18.655 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:49:18.655 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-11T21:49:18.670 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:49:18.670 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:49:18.670 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:49:18.670 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:49:19.672 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:49:19.672 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:49:19.741 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-03-11T21:49:19.776 DEBUG:teuthology.parallel:result is None 2024-03-11T21:49:25.154 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-11T21:49:40.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:49:40.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:49:40.502 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-11T21:49:40.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:49:40.904 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:49:40.904 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-11T21:49:40.918 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:49:40.918 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:49:40.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:49:40.918 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:49:41.920 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T21:49:41.920 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:49:41.990 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-03-11T21:49:42.052 DEBUG:teuthology.parallel:result is None 2024-03-11T21:49:42.053 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:49:42.060 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:49:42.060 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:49:42.063 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:49:42.065 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:49:42.081 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:49:42.177 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:49:42.246 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:49:42.246 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:49:42.249 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:49:42.251 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:49:42.410 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:49:42.417 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:49:42.417 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:49:42.420 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:49:42.422 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:49:42.450 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:49:42.452 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:49:42.453 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:49:42.462 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:49:42.466 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:49:42.467 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:49:42.468 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:49:42.476 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:49:42.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:49:42.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:49:42.514 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:49:42.545 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:49:42.554 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:49:42.554 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:49:42.583 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:49:42.586 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:49:42.601 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:49:42.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:49:42.773 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:49:42.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:49:42.857 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:49:42.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:49:42.925 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:49:42.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:49:43.028 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:49:43.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:49:43.157 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:49:43.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:49:43.227 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:49:43.280 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:49:43.358 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:49:43.413 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:49:43.473 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:49:43.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:49:43.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-11T21:49:43.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-03-11T21:49:43.562 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-03-11T21:49:43.616 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:49:43.617 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:49:43.675 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:49:44.179 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:49:44.187 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:49:44.187 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:49:44.221 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:49:44.260 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T21:49:44.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-03-11T21:49:44.304 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-03-11T21:49:44.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-03-11T21:49:44.356 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-03-11T21:49:44.358 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-03-11T21:49:44.610 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-03-11T21:49:44.612 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:49:44.612 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-03-11T21:49:44.633 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-03-11T21:49:44.634 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-03-11T21:49:44.653 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-03-11T21:49:44.653 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-03-11T21:49:44.741 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-03-11T21:49:44.741 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-03-11T21:49:44.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:49:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:49:44.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:49:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:49:44.801 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:49:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:49:44.832 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:49:44.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:49:44.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:49:44.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:49:44.968 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:49:44.979 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:49:44.996 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:49:44.997 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi176.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:53:56.983 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-03-11T21:53:56.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:53:56.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:53:57.076 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-11T21:53:57.161 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-11T21:53:57.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-11T21:53:57.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:53:57.250 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-11T21:53:57.328 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-11T21:53:57.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:53:57.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:53:57.415 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-11T21:53:57.494 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-11T21:53:57.495 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:53:57.506 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:53:57.507 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:53:57.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:53:57.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:53:57.510 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:53:57.512 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:53:57.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:53:57.540 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:53:57.541 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:53:57.557 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:53:57.558 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:53:57.568 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:53:57.586 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:53:57.587 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-03-11T21:53:57.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-03-11T21:53:57.600 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-03-11T21:53:57.602 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-03-11T21:53:57.616 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-03-11T21:53:57.617 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:53:57.618 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:53:57.630 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-03-11T21:53:57.632 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:53:57.635 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:53:57.647 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:53:57.663 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:53:57.688 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:53:57.688 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:53:57.692 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:53:57.692 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-11T21:53:57.692 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.692 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.692 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.692 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.693 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:53:57.693 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-11T21:53:57.693 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.693 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.694 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.694 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.716 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:53:57.719 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:53:57.719 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-11T21:53:57.719 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.719 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.719 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.719 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:53:57.720 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T21:53:57.730 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T21:53:57.730 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi151.front.sepia.ceph.com 2024-03-11T21:53:57.730 DEBUG:teuthology.orchestra.run.smithi151:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:53:57.731 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi176.front.sepia.ceph.com 2024-03-11T21:53:57.731 DEBUG:teuthology.orchestra.run.smithi176:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:53:57.731 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi189.front.sepia.ceph.com 2024-03-11T21:53:57.731 DEBUG:teuthology.orchestra.run.smithi189:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:53:58.319 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:49 ago on Mon 11 Mar 2024 09:53:09 PM UTC. 2024-03-11T21:53:58.322 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:49 ago on Mon 11 Mar 2024 09:53:09 PM UTC. 2024-03-11T21:53:58.345 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 09:53:10 PM UTC. 2024-03-11T21:53:58.466 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-11T21:53:58.469 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-11T21:53:58.470 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-11T21:53:58.470 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-11T21:53:58.470 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-11T21:53:58.470 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-11T21:53:58.470 INFO:teuthology.orchestra.run.smithi176.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:53:58.471 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:53:58.471 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-11T21:53:58.471 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-11T21:53:58.471 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-11T21:53:58.471 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:53:58.479 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.1 M 2024-03-11T21:53:58.480 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 9.8 M 2024-03-11T21:53:58.480 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:53:58.486 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-11T21:53:58.487 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-11T21:53:58.487 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-03-11T21:53:58.487 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:53:58.489 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-03-11T21:53:58.489 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 9.8 M 2024-03-11T21:53:58.489 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-11T21:53:58.491 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-11T21:53:58.502 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-11T21:53:58.502 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-11T21:53:58.502 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-11T21:53:58.502 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-11T21:53:58.503 INFO:teuthology.orchestra.run.smithi189.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:53:58.503 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:53:58.503 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-11T21:53:58.503 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-11T21:53:58.503 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-11T21:53:58.503 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:53:58.512 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.1 M 2024-03-11T21:53:58.512 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 9.8 M 2024-03-11T21:53:58.512 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-11T21:53:59.092 INFO:teuthology.orchestra.run.smithi176.stdout:runc-1.1.12-2.el9.x86_64.rpm 6.3 MB/s | 3.1 MB 00:00 2024-03-11T21:53:59.092 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:53:59.093 INFO:teuthology.orchestra.run.smithi176.stdout:Total 5.1 MB/s | 3.1 MB 00:00 2024-03-11T21:53:59.128 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-11T21:53:59.134 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-11T21:53:59.134 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-11T21:53:59.212 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-11T21:53:59.213 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-11T21:53:59.417 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-11T21:53:59.512 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:53:59.585 INFO:teuthology.orchestra.run.smithi151.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.2 MB/s | 3.1 MB 00:00 2024-03-11T21:53:59.586 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:53:59.586 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-11T21:53:59.623 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-11T21:53:59.630 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-11T21:53:59.630 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-11T21:53:59.711 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-11T21:53:59.712 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-11T21:53:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-11T21:54:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:00.077 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:00.408 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:00.408 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:54:00.408 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-11T21:54:00.408 INFO:teuthology.orchestra.run.smithi176.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:54:00.408 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-11T21:54:00.408 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-11T21:54:00.523 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:00.656 DEBUG:teuthology.parallel:result is None 2024-03-11T21:54:00.825 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:00.825 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:54:00.826 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-11T21:54:00.826 INFO:teuthology.orchestra.run.smithi151.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:54:00.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:54:00.826 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-11T21:54:01.056 DEBUG:teuthology.parallel:result is None 2024-03-11T21:54:03.638 INFO:teuthology.orchestra.run.smithi189.stdout:runc-1.1.12-2.el9.x86_64.rpm 641 kB/s | 3.1 MB 00:04 2024-03-11T21:54:03.639 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:54:03.639 INFO:teuthology.orchestra.run.smithi189.stdout:Total 625 kB/s | 3.1 MB 00:05 2024-03-11T21:54:03.676 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-11T21:54:03.682 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-11T21:54:03.683 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-11T21:54:03.765 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-11T21:54:03.765 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-11T21:54:03.985 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-11T21:54:04.083 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:04.728 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:05.033 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:54:05.033 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:54:05.033 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-11T21:54:05.033 INFO:teuthology.orchestra.run.smithi189.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:54:05.033 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:54:05.033 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-11T21:54:05.242 DEBUG:teuthology.parallel:result is None 2024-03-11T21:54:05.242 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T21:54:05.253 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T21:54:05.253 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:54:05.253 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:54:05.273 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:54:05.273 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:53:55.950034777 +0000 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:53:33.443331232 +0000 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:53:33.443331232 +0000 2024-03-11T21:54:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:53:33.443331232 +0000 2024-03-11T21:54:05.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:54:05.400 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:54:05.400 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:54:05.400 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000286254 s, 1.8 MB/s 2024-03-11T21:54:05.401 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:54:05.458 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:53:55.950034777 +0000 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:53:33.784326740 +0000 2024-03-11T21:54:05.514 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:53:33.784326740 +0000 2024-03-11T21:54:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:53:33.784326740 +0000 2024-03-11T21:54:05.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:54:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:54:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:54:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000273111 s, 1.9 MB/s 2024-03-11T21:54:05.583 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:54:05.640 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:53:55.950034777 +0000 2024-03-11T21:54:05.696 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:53:34.125322249 +0000 2024-03-11T21:54:05.697 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:53:34.125322249 +0000 2024-03-11T21:54:05.697 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:53:34.125322249 +0000 2024-03-11T21:54:05.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:54:05.762 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:54:05.762 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:54:05.762 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000203876 s, 2.5 MB/s 2024-03-11T21:54:05.763 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:54:05.820 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:53:55.950034777 +0000 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:53:34.469317718 +0000 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:53:34.469317718 +0000 2024-03-11T21:54:05.876 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:53:34.469317718 +0000 2024-03-11T21:54:05.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:54:05.942 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:54:05.942 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:54:05.943 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000279452 s, 1.8 MB/s 2024-03-11T21:54:05.943 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:54:05.999 DEBUG:teuthology.orchestra.run.smithi151:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:54:06.161 INFO:teuthology.orchestra.run.smithi151.stdout:loop 2024-03-11T21:54:06.163 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_1... 2024-03-11T21:54:06.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:54:06.200 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-11T21:54:06.237 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:54:06.274 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_2... 2024-03-11T21:54:06.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:54:06.355 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-11T21:54:06.394 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:54:06.431 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_3... 2024-03-11T21:54:06.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:54:06.508 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-11T21:54:06.547 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:54:06.590 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_4... 2024-03-11T21:54:06.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:54:06.671 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-11T21:54:06.711 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:54:06.742 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:54:06.742 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:54:06.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme list 2024-03-11T21:54:06.864 INFO:teuthology.orchestra.run.smithi151.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:54:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:54:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme0n1 CVFT623300JJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T21:54:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme1n1 4c7c7e7b0eb6023c53b1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme2n1 515856c017f0909c15a7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme3n1 4413c8b1878e15261397 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme4n1 22ae7e8451852b8aba00 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:06.866 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:54:06.866 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:54:06.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/scratch_devs 2024-03-11T21:54:06.934 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:54:06.934 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:54:06.952 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:54:06.953 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-03-11T21:54:07.010 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:53:55.968512512 +0000 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:53:34.201726462 +0000 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:53:34.201726462 +0000 2024-03-11T21:54:07.011 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:53:34.201726462 +0000 2024-03-11T21:54:07.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:54:07.080 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:54:07.080 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:54:07.080 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000275233 s, 1.9 MB/s 2024-03-11T21:54:07.081 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:54:07.139 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-03-11T21:54:07.196 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:54:07.196 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:07.196 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T21:54:07.196 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:07.196 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:07.196 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:53:55.968512512 +0000 2024-03-11T21:54:07.197 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:53:34.547723061 +0000 2024-03-11T21:54:07.197 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:53:34.547723061 +0000 2024-03-11T21:54:07.197 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:53:34.547723061 +0000 2024-03-11T21:54:07.197 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:54:07.263 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:54:07.263 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:54:07.263 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000240583 s, 2.1 MB/s 2024-03-11T21:54:07.264 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:54:07.320 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-03-11T21:54:07.376 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:53:55.968512512 +0000 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:53:34.891719679 +0000 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:53:34.891719679 +0000 2024-03-11T21:54:07.377 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:53:34.891719679 +0000 2024-03-11T21:54:07.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:54:07.443 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:54:07.443 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:54:07.444 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000192463 s, 2.7 MB/s 2024-03-11T21:54:07.445 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:54:07.501 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:53:55.968512512 +0000 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:53:35.231716338 +0000 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:53:35.231716338 +0000 2024-03-11T21:54:07.557 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:53:35.231716338 +0000 2024-03-11T21:54:07.558 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:54:07.623 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:54:07.623 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:54:07.623 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000173214 s, 3.0 MB/s 2024-03-11T21:54:07.624 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:54:07.680 DEBUG:teuthology.orchestra.run.smithi176:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:54:07.836 INFO:teuthology.orchestra.run.smithi176.stdout:loop 2024-03-11T21:54:07.837 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_1... 2024-03-11T21:54:07.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:54:07.875 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-11T21:54:07.911 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:54:07.940 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_2... 2024-03-11T21:54:07.941 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:54:08.021 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-11T21:54:08.060 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:54:08.089 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_3... 2024-03-11T21:54:08.089 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:54:08.167 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-11T21:54:08.206 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:54:08.235 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_4... 2024-03-11T21:54:08.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:54:08.314 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-11T21:54:08.354 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:54:08.384 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:54:08.384 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:54:08.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme list 2024-03-11T21:54:08.505 INFO:teuthology.orchestra.run.smithi176.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:54:08.505 INFO:teuthology.orchestra.run.smithi176.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:54:08.506 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme0n1 CVFT62330099400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T21:54:08.506 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme1n1 3419fd1895239dbce5f3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:08.506 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme2n1 e2a76087d578020c7e93 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:08.506 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme3n1 48fe081f6ae86fd6b4c6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:08.506 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme4n1 5c0009d97f35ee05ff2f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:08.507 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:54:08.507 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:54:08.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/scratch_devs 2024-03-11T21:54:08.574 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:54:08.574 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:54:08.592 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:54:08.592 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:53:56.031680501 +0000 2024-03-11T21:54:08.650 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:53:35.334107297 +0000 2024-03-11T21:54:08.651 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:53:35.334107297 +0000 2024-03-11T21:54:08.651 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:53:35.334107297 +0000 2024-03-11T21:54:08.651 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:54:08.720 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:54:08.720 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:54:08.720 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000286186 s, 1.8 MB/s 2024-03-11T21:54:08.721 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:54:08.778 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-03-11T21:54:08.835 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:54:08.835 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:08.835 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T21:54:08.835 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:08.835 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:08.835 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:53:56.031680501 +0000 2024-03-11T21:54:08.836 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:53:35.687100018 +0000 2024-03-11T21:54:08.836 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:53:35.687100018 +0000 2024-03-11T21:54:08.836 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:53:35.687100018 +0000 2024-03-11T21:54:08.836 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:54:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:54:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:54:08.904 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000212806 s, 2.4 MB/s 2024-03-11T21:54:08.905 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:54:08.962 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-03-11T21:54:09.017 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:53:56.031680501 +0000 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:53:36.032092904 +0000 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:53:36.032092904 +0000 2024-03-11T21:54:09.018 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:53:36.032092904 +0000 2024-03-11T21:54:09.018 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:54:09.085 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:54:09.085 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:54:09.085 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000221408 s, 2.3 MB/s 2024-03-11T21:54:09.086 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:54:09.143 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-03-11T21:54:09.199 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:53:56.031680501 +0000 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:53:36.370085934 +0000 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:53:36.370085934 +0000 2024-03-11T21:54:09.200 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:53:36.370085934 +0000 2024-03-11T21:54:09.200 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:54:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:54:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:54:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000236084 s, 2.2 MB/s 2024-03-11T21:54:09.269 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:54:09.325 DEBUG:teuthology.orchestra.run.smithi189:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:54:09.484 INFO:teuthology.orchestra.run.smithi189.stdout:loop 2024-03-11T21:54:09.485 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_1... 2024-03-11T21:54:09.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:54:09.523 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-11T21:54:09.561 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:54:09.590 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_2... 2024-03-11T21:54:09.590 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:54:09.677 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-11T21:54:09.716 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:54:09.745 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_3... 2024-03-11T21:54:09.746 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:54:09.829 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-11T21:54:09.867 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:54:09.897 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_4... 2024-03-11T21:54:09.897 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:54:09.977 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-11T21:54:10.018 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:54:10.047 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:54:10.047 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:54:10.102 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme list 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme0n1 CVFT6232004L400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme1n1 bc90b600a98798b48ce1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme2n1 e77c53556cf9c79f9bf4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme3n1 763fa3016cae2b9de0e8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:10.170 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme4n1 caf5c065b726d7e361b3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:54:10.171 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:54:10.171 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:54:10.171 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/scratch_devs 2024-03-11T21:54:10.239 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T21:54:10.328 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T21:54:10.328 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:54:10.329 INFO:tasks.cephadm:Cluster fsid is e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:54:10.329 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T21:54:10.329 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.151', 'mon.b': '172.21.15.176', 'mon.c': '172.21.15.189'} 2024-03-11T21:54:10.329 INFO:tasks.cephadm:First mon is mon.a on smithi151 2024-03-11T21:54:10.329 INFO:tasks.cephadm:First mgr is y 2024-03-11T21:54:10.329 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T21:54:10.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-03-11T21:54:10.362 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-03-11T21:54:10.395 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-03-11T21:54:10.422 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T21:54:10.422 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:54:10.711 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T21:54:11.006 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:54:11.008 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T21:54:11.008 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:11.463 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:54 /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:11.463 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:11.914 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:54 /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:11.914 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:12.340 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:54 /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:12.341 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:12.360 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:12.379 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:54:12.409 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T21:54:12.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:54:12.411 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:54:12.421 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:54:13.187 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:54:13.268 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:54:13.270 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-03-11T21:54:38.785 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-03-11T21:55:08.459 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-11T21:55:08.460 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:55:08.460 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:55:08.460 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2024-03-11T21:55:08.460 INFO:teuthology.orchestra.run.smithi176.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:55:08.460 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-11T21:55:08.460 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-11T21:55:09.451 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-11T21:55:09.452 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:55:09.452 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:55:09.452 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-03-11T21:55:09.452 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:55:09.452 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-03-11T21:55:09.452 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-11T21:55:09.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-03-11T21:55:09.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-03-11T21:55:09.552 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-03-11T21:55:09.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-03-11T21:55:09.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-03-11T21:55:09.644 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-03-11T21:55:09.673 INFO:tasks.cephadm:Writing seed config... 2024-03-11T21:55:09.674 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T21:55:09.674 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T21:55:09.674 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-11T21:55:09.674 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-11T21:55:09.675 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T21:55:09.675 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T21:55:09.675 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T21:55:09.675 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T21:55:09.675 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T21:55:09.675 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T21:55:09.676 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:55:09.676 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T21:55:09.693 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e42c6b1c-dff1-11ee-95c8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T21:55:09.693 DEBUG:teuthology.orchestra.run.smithi151:mon.a> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T21:55:09.737 DEBUG:teuthology.orchestra.run.smithi151:mgr.y> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T21:55:09.780 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T21:55:09.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.151 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:55:09.971 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:55:09.971 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', 'e42c6b1c-dff1-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.151', '--skip-admin-label'] 2024-03-11T21:55:09.997 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:55:09.998 INFO:teuthology.orchestra.run.smithi151.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T21:55:09.999 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying podman|docker is present... 2024-03-11T21:55:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:55:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying lvm2 is present... 2024-03-11T21:55:10.022 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying time synchronization is in place... 2024-03-11T21:55:10.029 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:55:10.029 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:55:10.035 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:55:10.035 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-03-11T21:55:10.042 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-03-11T21:55:10.047 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-03-11T21:55:10.048 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:55:10.048 INFO:teuthology.orchestra.run.smithi151.stdout:Repeating the final host check... 2024-03-11T21:55:10.071 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:55:10.071 INFO:teuthology.orchestra.run.smithi151.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T21:55:10.071 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl is present 2024-03-11T21:55:10.071 INFO:teuthology.orchestra.run.smithi151.stdout:lvcreate is present 2024-03-11T21:55:10.077 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:55:10.077 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:55:10.083 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:55:10.083 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-03-11T21:55:10.089 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-03-11T21:55:10.095 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-03-11T21:55:10.095 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:55:10.096 INFO:teuthology.orchestra.run.smithi151.stdout:Host looks OK 2024-03-11T21:55:10.096 INFO:teuthology.orchestra.run.smithi151.stdout:Cluster fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:10.096 INFO:teuthology.orchestra.run.smithi151.stdout:Acquiring lock 140537519667808 on /run/cephadm/e42c6b1c-dff1-11ee-95c8-87774f69a715.lock 2024-03-11T21:55:10.096 INFO:teuthology.orchestra.run.smithi151.stdout:Lock 140537519667808 acquired on /run/cephadm/e42c6b1c-dff1-11ee-95c8-87774f69a715.lock 2024-03-11T21:55:10.096 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 3300 ... 2024-03-11T21:55:10.096 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 6789 ... 2024-03-11T21:55:10.097 INFO:teuthology.orchestra.run.smithi151.stdout:Base mon IP(s) is [172.21.15.151:3300, 172.21.15.151:6789], mon addrv is [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-03-11T21:55:10.099 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.151 metric 100 2024-03-11T21:55:10.099 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.151 metric 100 2024-03-11T21:55:10.102 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T21:55:10.102 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T21:55:10.102 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-03-11T21:55:10.104 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T21:55:10.104 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T21:55:10.104 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:55:10.105 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T21:55:10.105 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6f63/64 scope link 2024-03-11T21:55:10.105 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:55:10.105 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-03-11T21:55:10.105 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-03-11T21:55:10.105 INFO:teuthology.orchestra.run.smithi151.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T21:55:10.106 INFO:teuthology.orchestra.run.smithi151.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T21:55:10.106 INFO:teuthology.orchestra.run.smithi151.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:55:10.519 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T21:55:11.392 INFO:teuthology.orchestra.run.smithi151.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T21:55:11.392 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T21:55:11.392 INFO:teuthology.orchestra.run.smithi151.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T21:55:12.317 INFO:teuthology.orchestra.run.smithi151.stdout:stat: stdout 167 167 2024-03-11T21:55:12.317 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial keys... 2024-03-11T21:55:13.282 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQDAfe9lqkZZJxAA6MHak80iXl59bU8XOwK5Fw== 2024-03-11T21:55:14.256 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQDBfe9lf4HOJBAASUyBDS9qSk23+2s80+8CPA== 2024-03-11T21:55:15.321 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQDCfe9l3T+4JhAA4Am9mUbMGGVcXRfXOcpyLw== 2024-03-11T21:55:15.322 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial monmap... 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool for a [v2:172.21.15.151:3300,v1:172.21.15.151:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = quincy 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: set fsid to e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:55:16.221 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mon... 2024-03-11T21:55:17.169 INFO:teuthology.orchestra.run.smithi151.stdout:create mon.a on 2024-03-11T21:55:17.520 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T21:55:17.701 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715.target. 2024-03-11T21:55:17.701 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715.target. 2024-03-11T21:55:17.957 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:55:17.957 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T21:55:18.120 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715.target.wants/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@.service. 2024-03-11T21:55:18.672 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:55:18.673 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-03-11T21:55:18.679 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:55:18.679 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-03-11T21:55:18.679 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-03-11T21:55:18.679 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:55:18.679 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon to start... 2024-03-11T21:55:18.679 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon... 2024-03-11T21:55:19.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T21:55:19.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout id: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:19.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T21:55:19.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:19.944 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout services: 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.585405s) 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout data: 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:mon is available 2024-03-11T21:55:19.945 INFO:teuthology.orchestra.run.smithi151.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T21:55:20.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:20 smithi151 ceph-mon[25286]: from='client.? 172.21.15.151:0/4007962733' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:55:21.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:21.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:55:21.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:21.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-03-11T21:55:21.067 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:55:21.068 INFO:teuthology.orchestra.run.smithi151.stdout:Generating new minimal ceph.conf... 2024-03-11T21:55:21.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:21 smithi151 ceph-mon[25286]: from='client.? 172.21.15.151:0/84081368' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:55:22.124 INFO:teuthology.orchestra.run.smithi151.stdout:Restarting the monitor... 2024-03-11T21:55:22.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:22 smithi151 systemd[1]: Stopping Ceph mon.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:55:22.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:22 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a[25263]: 2024-03-11T21:55:22.400+0000 7f41d1126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:55:22.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:22 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a[25263]: 2024-03-11T21:55:22.400+0000 7f41d1126700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T21:55:22.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:22 smithi151 podman[25618]: 2024-03-11 21:55:22.523918214 +0000 UTC m=+0.226566469 container died 741056370d93155e3334b0ce3897ae1a85b9796888d3fac9376e497072145e04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:55:23.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:22 smithi151 podman[25618]: 2024-03-11 21:55:22.873137711 +0000 UTC m=+0.575785958 container cleanup 741056370d93155e3334b0ce3897ae1a85b9796888d3fac9376e497072145e04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T21:55:23.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:22 smithi151 bash[25618]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:55:23.476 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 podman[25637]: 2024-03-11 21:55:23.141204048 +0000 UTC m=+0.613323118 container remove 741056370d93155e3334b0ce3897ae1a85b9796888d3fac9376e497072145e04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True) 2024-03-11T21:55:23.476 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T21:55:23.477 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 systemd[1]: Stopped Ceph mon.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:55:23.477 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 systemd[1]: Starting Ceph mon.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:55:23.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 podman[25735]: 2024-03-11 21:55:23.483796687 +0000 UTC m=+0.096038356 container create d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T21:55:23.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 podman[25735]: 2024-03-11 21:55:23.407139351 +0000 UTC m=+0.019380992 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 podman[25735]: 2024-03-11 21:55:23.661866568 +0000 UTC m=+0.274108196 container init d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 podman[25735]: 2024-03-11 21:55:23.667354763 +0000 UTC m=+0.279596394 container start d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: pidfile_write: ignore empty --pid-file 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: load: jerasure load: lrc 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Git sha 0 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: DB SUMMARY 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: DB Session ID: 7L581JV2A9MWUKQH95XI 2024-03-11T21:55:23.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84645 ; 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.env: 0x55699322eae0 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:55:23.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.info_log: 0x5569944f58c0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.statistics: (nil) 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.use_fsync: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:55:23.749 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.db_log_dir: 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.wal_dir: 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.write_buffer_manager: 0x5569944c9400 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:55:23.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:55:23.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:55:23.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:55:23.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.unordered_write: 0 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.row_cache: None 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.wal_filter: None 2024-03-11T21:55:23.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.wal_compression: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:55:23.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:55:23.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_open_files: -1 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Compression algorithms supported: 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kZSTD supported: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:55:23.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.merge_operator: 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_filter: None 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5569944f59c0) 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:55:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_type: 0 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_index_type: 0 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_shortening: 1 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: checksum: 4 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: no_block_cache: 0 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache: 0x5569944f9090 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_options: 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: capacity : 536870912 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_shard_bits : 4 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: strict_capacity_limit : 0 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:55:23.757 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_compressed: (nil) 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: persistent_cache: (nil) 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size: 4096 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size_deviation: 10 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_restart_interval: 16 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_block_restart_interval: 1 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: metadata_block_size: 4096 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: partition_filters: 0 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: use_delta_encoding: 1 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: filter_policy: bloomfilter 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: whole_key_filtering: 1 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: verify_compression: 0 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: format_version: 5 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_index_compression: 1 2024-03-11T21:55:23.758 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_align: 0 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_auto_readahead_size: 262144 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout: prepopulate_block_cache: 0 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression: NoCompression 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.num_levels: 7 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:55:23.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:55:23.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:55:23.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:55:23.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:55:23.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.ttl: 2592000 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:55:23.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ab8fca6a-8085-4f68-b281-2f51725f9db6 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194123696642, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194123697478, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79457, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73766, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710194123, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ab8fca6a-8085-4f68-b281-2f51725f9db6", "db_session_id": "7L581JV2A9MWUKQH95XI", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194123697561, "job": 1, "event": "recovery_finished"} 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5569945ea000 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: DB pointer 0x5569945d4000 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** DB Stats ** 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:55:23.765 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: L0 2/0 81.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Sum 2/0 81.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:55:23.766 INFO:journalctl@ceph.mon.a.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative compaction: 0.00 GB write, 27.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval compaction: 0.00 GB write, 27.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache BinnedLRUCache@0x5569944f9090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:55:23.767 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: starting mon.a rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???) e1 preinit fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).mds e1 new map 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).mds e1 print_map 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: e1 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: legacy client fscid: -1 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout: No filesystems configured 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:55:23.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:55:23.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T21:55:23.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 bash[25735]: d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 2024-03-11T21:55:23.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 systemd[1]: Started Ceph mon.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:55:23.769 INFO:teuthology.orchestra.run.smithi151.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T21:55:24.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:55:24.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: monmap e1: 1 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:55:24.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: fsmap 2024-03-11T21:55:24.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:55:24.006 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:23 smithi151 ceph-mon[25773]: mgrmap e1: no daemons active 2024-03-11T21:55:24.898 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T21:55:24.920 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:55:24.920 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mgr... 2024-03-11T21:55:24.921 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T21:55:24.921 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T21:55:25.154 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:55:25.155 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T21:55:25.312 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715.target.wants/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@.service. 2024-03-11T21:55:25.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:25 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/3968709272' entity='client.admin' 2024-03-11T21:55:25.545 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 systemd[1]: Starting Ceph mgr.y for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:55:25.804 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 podman[26010]: 2024-03-11 21:55:25.454886254 +0000 UTC m=+0.017796530 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:55:25.805 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 podman[26010]: 2024-03-11 21:55:25.563445007 +0000 UTC m=+0.126355278 container create 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T21:55:25.805 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 podman[26010]: 2024-03-11 21:55:25.791088746 +0000 UTC m=+0.353999030 container init 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T21:55:25.805 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 podman[26010]: 2024-03-11 21:55:25.796321987 +0000 UTC m=+0.359232259 container start 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:55:25.908 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:55:25.908 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-03-11T21:55:25.916 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:55:25.916 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-03-11T21:55:25.916 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-03-11T21:55:25.916 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:55:25.926 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:55:25.926 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-03-11T21:55:25.934 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:55:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-03-11T21:55:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-03-11T21:55:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T21:55:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr to start... 2024-03-11T21:55:25.935 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr... 2024-03-11T21:55:26.164 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 bash[26010]: 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 2024-03-11T21:55:26.164 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:25 smithi151 systemd[1]: Started Ceph mgr.y for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:55:26.459 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:26.368+0000 7f3a3e297200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:55:26.459 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:26.465+0000 7f3a3e297200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:55:26.757 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:26 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/3104517540' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:55:27.055 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "e42c6b1c-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:55:27.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:55:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:55:27.061 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:55:27.062 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:55:18.629803+0000", 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:55:27.063 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (1/15)... 2024-03-11T21:55:27.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.122+0000 7f3a3e297200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:55:27.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.214+0000 7f3a3e297200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:55:27.784 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.355+0000 7f3a3e297200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:55:27.784 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.450+0000 7f3a3e297200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:55:27.785 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.531+0000 7f3a3e297200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:55:27.785 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.790+0000 7f3a3e297200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:55:28.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.899+0000 7f3a3e297200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:28.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:27.983+0000 7f3a3e297200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:55:28.323 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:28 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:28.076+0000 7f3a3e297200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:55:28.323 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:28 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:28.252+0000 7f3a3e297200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:55:29.478 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:29.225+0000 7f3a3e297200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:55:29.479 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:29.369+0000 7f3a3e297200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:55:29.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:29 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/996936198' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:55:29.803 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:29.716+0000 7f3a3e297200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:55:29.803 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:29.809+0000 7f3a3e297200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:55:30.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:30.005+0000 7f3a3e297200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:30.104 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:30.104 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "e42c6b1c-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:30.105 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:55:30.106 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:55:30.107 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:55:30.107 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:55:30.107 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:55:30.107 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:55:30.110 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.110 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:55:30.110 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:55:30.110 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:55:30.111 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:55:18.629803+0000", 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:55:30.112 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (2/15)... 2024-03-11T21:55:30.467 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:30.190+0000 7f3a3e297200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:55:30.727 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:30.473+0000 7f3a3e297200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:30.727 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:30.733+0000 7f3a3e297200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:31.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:30.966+0000 7f3a3e297200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:55:32.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:31 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:31.757+0000 7f3a3e297200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:55:32.553 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:32 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:32.345+0000 7f3a3e297200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:55:32.553 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:32 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:32.443+0000 7f3a3e297200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:55:32.804 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:32 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:32.666+0000 7f3a3e297200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:55:32.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:32 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/3401145517' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:55:32.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:32 smithi151 ceph-mon[25773]: Activating manager daemon y 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "e42c6b1c-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:55:33.179 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:55:33.180 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:55:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:55:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:55:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:55:33.184 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:55:33.185 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:55:18.629803+0000", 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:55:33.186 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (3/15)... 2024-03-11T21:55:34.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: mgrmap e2: y(active, starting, since 0.00287369s) 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: Manager daemon y is now available 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' 2024-03-11T21:55:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:33 smithi151 ceph-mon[25773]: from='mgr.14100 172.21.15.151:0/460340106' entity='mgr.y' 2024-03-11T21:55:35.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:34 smithi151 ceph-mon[25773]: mgrmap e3: y(active, since 1.01152s) 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "e42c6b1c-dff1-11ee-95c8-87774f69a715", 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:55:36.469 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:55:36.470 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:55:36.471 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:55:36.474 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:55:36.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:55:18.629803+0000", 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:55:36.476 INFO:teuthology.orchestra.run.smithi151.stdout:mgr is available 2024-03-11T21:55:36.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:36 smithi151 ceph-mon[25773]: mgrmap e4: y(active, since 3s) 2024-03-11T21:55:36.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:36 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4002955997' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:55:37.643 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:55:37.644 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:55:37.645 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling cephadm module... 2024-03-11T21:55:37.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:37 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/90379347' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:55:37.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:37 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/90379347' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:55:39.005 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:38 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1992552913' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T21:55:39.005 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:38 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ignoring --setuser ceph since I am not root 2024-03-11T21:55:39.005 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:38 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ignoring --setgroup ceph since I am not root 2024-03-11T21:55:39.255 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:39 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:39.260+0000 7fcef7476200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:55:39.514 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:39 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:39.364+0000 7fcef7476200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:55:39.974 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:39 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1992552913' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T21:55:39.974 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:39 smithi151 ceph-mon[25773]: mgrmap e5: y(active, since 6s) 2024-03-11T21:55:40.253 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.049+0000 7fcef7476200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:55:40.254 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.143+0000 7fcef7476200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:55:40.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.288+0000 7fcef7476200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:55:40.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.384+0000 7fcef7476200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:55:40.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.467+0000 7fcef7476200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:55:40.558 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:55:40.558 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T21:55:40.558 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:55:40.559 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:55:40.559 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:55:40.559 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:55:40.559 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-03-11T21:55:40.559 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 5... 2024-03-11T21:55:40.809 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1167974735' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:55:40.810 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.760+0000 7fcef7476200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:55:41.074 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.892+0000 7fcef7476200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:41.074 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:40.981+0000 7fcef7476200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:55:41.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:41 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:41.080+0000 7fcef7476200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:55:41.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:41 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:41.258+0000 7fcef7476200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:55:42.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:42 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:42.213+0000 7fcef7476200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:55:42.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:42 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:42.344+0000 7fcef7476200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:55:42.959 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:42 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:42.678+0000 7fcef7476200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:55:42.959 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:42 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:42.768+0000 7fcef7476200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:55:42.959 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:42 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:42.965+0000 7fcef7476200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:43.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:43 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:43.151+0000 7fcef7476200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:55:43.689 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:43 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:43.435+0000 7fcef7476200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:55:43.689 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:43 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:43.695+0000 7fcef7476200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:55:44.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:43 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:43.930+0000 7fcef7476200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:55:45.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:44 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:44.731+0000 7fcef7476200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:55:45.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:45.303+0000 7fcef7476200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:55:45.554 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:45.402+0000 7fcef7476200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:55:46.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:55:45.619+0000 7fcef7476200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: Active manager daemon y restarted 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: Activating manager daemon y 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: mgrmap e6: y(active, starting, since 0.00337294s) 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: Manager daemon y is now available 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:45 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: Found migration_current of "None". Setting to last migration. 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: mgrmap e7: y(active, since 1.00549s) 2024-03-11T21:55:46.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:46 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:47.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:55:47.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T21:55:47.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:55:47.166 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:55:47.166 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 5 is available 2024-03-11T21:55:47.167 INFO:teuthology.orchestra.run.smithi151.stdout:Setting orchestrator backend to cephadm... 2024-03-11T21:55:47.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:47 smithi151 ceph-mon[25773]: [11/Mar/2024:21:55:46] ENGINE Bus STARTING 2024-03-11T21:55:47.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:47 smithi151 ceph-mon[25773]: [11/Mar/2024:21:55:46] ENGINE Serving on http://172.21.15.151:8765 2024-03-11T21:55:47.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:47 smithi151 ceph-mon[25773]: [11/Mar/2024:21:55:46] ENGINE Serving on https://172.21.15.151:7150 2024-03-11T21:55:47.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:47 smithi151 ceph-mon[25773]: [11/Mar/2024:21:55:46] ENGINE Bus STARTED 2024-03-11T21:55:47.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:47 smithi151 ceph-mon[25773]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:55:47.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:47 smithi151 ceph-mon[25773]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:55:48.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:48 smithi151 ceph-mon[25773]: mgrmap e8: y(active, since 2s) 2024-03-11T21:55:48.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:48 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:48.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:48 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:49.505 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T21:55:49.505 INFO:teuthology.orchestra.run.smithi151.stdout:Generating ssh key... 2024-03-11T21:55:49.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:49 smithi151 ceph-mon[25773]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:50.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-mon[25773]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:50.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:50.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: Generating public/private rsa key pair. 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: Your identification has been saved in /tmp/tmpd1zsraqn/key. 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: Your public key has been saved in /tmp/tmpd1zsraqn/key.pub. 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: The key fingerprint is: 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: SHA256:DNHTviwaLtpqmndsHFzM4FZNcUBXRfb81J8gUhM7pw4 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: The key's randomart image is: 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: +---[RSA 3072]----+ 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | .==+o=oo+ | 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | . ..+oo o. o.| 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | . =. + + o =| 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | o +o o = ..+| 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | o . SE o .o| 2024-03-11T21:55:50.911 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | o . . = | 2024-03-11T21:55:50.912 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | o o o . . | 2024-03-11T21:55:50.912 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: | .o.* o | 2024-03-11T21:55:50.912 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: |o+o=.. | 2024-03-11T21:55:50.912 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:55:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: +----[SHA256]-----+ 2024-03-11T21:55:51.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:51 smithi151 ceph-mon[25773]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:51.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:51 smithi151 ceph-mon[25773]: Generating ssh key... 2024-03-11T21:55:52.271 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:55:52.271 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:55:52.271 INFO:teuthology.orchestra.run.smithi151.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T21:55:52.271 INFO:teuthology.orchestra.run.smithi151.stdout:Adding host smithi151... 2024-03-11T21:55:52.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:52 smithi151 ceph-mon[25773]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:53.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:53 smithi151 ceph-mon[25773]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "addr": "172.21.15.151", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:53 smithi151 ceph-mon[25773]: Deploying cephadm binary to smithi151 2024-03-11T21:55:57.630 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Added host 'smithi151' with addr '172.21.15.151' 2024-03-11T21:55:57.631 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mon service... 2024-03-11T21:55:58.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:58 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:55:58.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:58 smithi151 ceph-mon[25773]: Added host smithi151 2024-03-11T21:55:58.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:58 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:55:58.874 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T21:55:58.875 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mgr service... 2024-03-11T21:55:59.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:59 smithi151 ceph-mon[25773]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:55:59.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:59 smithi151 ceph-mon[25773]: Saving service mon spec with placement count:5 2024-03-11T21:55:59.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:55:59 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:00.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:00 smithi151 ceph-mon[25773]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:00.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:00 smithi151 ceph-mon[25773]: Saving service mgr spec with placement count:2 2024-03-11T21:56:00.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:00 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:00.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:00 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:00.823 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T21:56:02.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:01 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:02.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:01 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/2218537586' entity='client.admin' 2024-03-11T21:56:03.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' 2024-03-11T21:56:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi151", "caps": []}]: dispatch 2024-03-11T21:56:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: from='mgr.14120 172.21.15.151:0/697178156' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi151", "caps": []}]': finished 2024-03-11T21:56:03.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:03 smithi151 ceph-mon[25773]: Deploying daemon agent.smithi151 on smithi151 2024-03-11T21:56:04.019 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling the dashboard module... 2024-03-11T21:56:04.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:04 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4060982762' entity='client.admin' 2024-03-11T21:56:05.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:05 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/71101601' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T21:56:05.799 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:05 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ignoring --setuser ceph since I am not root 2024-03-11T21:56:05.800 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:05 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ignoring --setgroup ceph since I am not root 2024-03-11T21:56:06.311 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:06 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:06.064+0000 7f70fe4f3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:56:06.311 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:06 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:06.163+0000 7f70fe4f3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:56:06.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:06 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/71101601' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T21:56:06.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:06 smithi151 ceph-mon[25773]: mgrmap e9: y(active, since 19s) 2024-03-11T21:56:07.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:06 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:06.825+0000 7f70fe4f3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:56:07.055 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:06 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:06.920+0000 7f70fe4f3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:56:07.340 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.065+0000 7f70fe4f3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:56:07.340 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.161+0000 7f70fe4f3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:56:07.341 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.245+0000 7f70fe4f3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:56:07.343 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:56:07.343 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T21:56:07.343 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:56:07.343 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:56:07.344 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:56:07.344 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:56:07.344 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-03-11T21:56:07.344 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 9... 2024-03-11T21:56:07.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1058443011' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:56:07.634 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.525+0000 7f70fe4f3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:56:07.634 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.640+0000 7f70fe4f3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:56:08.003 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.729+0000 7f70fe4f3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:56:08.003 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:07 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:07.827+0000 7f70fe4f3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:56:08.003 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:08 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:08.006+0000 7f70fe4f3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:56:09.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:08 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:08.969+0000 7f70fe4f3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:56:09.305 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:09 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:09.099+0000 7f70fe4f3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:56:09.712 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:09 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:09.434+0000 7f70fe4f3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:56:09.712 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:09 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:09.525+0000 7f70fe4f3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:56:09.712 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:09 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:09.718+0000 7f70fe4f3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:56:10.054 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:09 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:09.908+0000 7f70fe4f3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:56:10.459 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:10 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:10.195+0000 7f70fe4f3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:56:10.459 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:10 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:10.465+0000 7f70fe4f3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:56:10.804 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:10 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:10.700+0000 7f70fe4f3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:56:11.804 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:11 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:11.496+0000 7f70fe4f3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:56:12.373 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:12.066+0000 7f70fe4f3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:56:12.373 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:12.164+0000 7f70fe4f3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:56:12.373 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:56:12.379+0000 7f70fe4f3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:56:12.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: Active manager daemon y restarted 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: Activating manager daemon y 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: mgrmap e10: y(active, starting, since 0.00330112s) 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: Manager daemon y is now available 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:13.738 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:13 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:56:13.738 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:13 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:56:13.738 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:13 smithi151 ceph-mon[25773]: mgrmap e11: y(active, since 1.0058s) 2024-03-11T21:56:14.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:56:14.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T21:56:14.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:56:14.356 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:56:14.356 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 9 is available 2024-03-11T21:56:14.356 INFO:teuthology.orchestra.run.smithi151.stdout:Generating a dashboard self-signed certificate... 2024-03-11T21:56:14.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:14 smithi151 ceph-mon[25773]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:56:14.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:14 smithi151 ceph-mon[25773]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:56:15.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: [11/Mar/2024:21:56:13] ENGINE Bus STARTING 2024-03-11T21:56:15.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: [11/Mar/2024:21:56:13] ENGINE Serving on http://172.21.15.151:8765 2024-03-11T21:56:15.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: [11/Mar/2024:21:56:13] ENGINE Serving on https://172.21.15.151:7150 2024-03-11T21:56:15.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: [11/Mar/2024:21:56:13] ENGINE Bus STARTED 2024-03-11T21:56:15.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: mgrmap e12: y(active, since 2s) 2024-03-11T21:56:15.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:15.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:15.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:15.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi151", "caps": []}]: dispatch 2024-03-11T21:56:15.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:15.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:15.714 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T21:56:15.714 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial admin user... 2024-03-11T21:56:16.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:16 smithi151 ceph-mon[25773]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:16.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:16 smithi151 ceph-mon[25773]: Deploying daemon agent.smithi151 on smithi151 2024-03-11T21:56:17.196 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$k6wQDxnX0SKKGXY/qZPT.eIEAWhBbVc/yykG4zr8bDX2rgdntXZ56", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710194176, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T21:56:17.196 INFO:teuthology.orchestra.run.smithi151.stdout:Fetching dashboard port number... 2024-03-11T21:56:17.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:17 smithi151 ceph-mon[25773]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:17.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:17 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:18.337 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T21:56:18.346 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:56:18.346 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-03-11T21:56:18.353 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:56:18.353 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-03-11T21:56:18.353 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-03-11T21:56:18.353 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T21:56:18.355 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph Dashboard is now available at: 2024-03-11T21:56:18.355 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:18.355 INFO:teuthology.orchestra.run.smithi151.stdout: URL: https://smithi151.front.sepia.ceph.com:8443/ 2024-03-11T21:56:18.356 INFO:teuthology.orchestra.run.smithi151.stdout: User: admin 2024-03-11T21:56:18.356 INFO:teuthology.orchestra.run.smithi151.stdout: Password: waou5d2hg2 2024-03-11T21:56:18.356 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:18.357 INFO:teuthology.orchestra.run.smithi151.stdout:Saving cluster configuration to /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config directory 2024-03-11T21:56:18.357 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling autotune for osd_memory_target 2024-03-11T21:56:18.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:18 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/2892028285' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T21:56:19.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:19 smithi151 ceph-mon[25773]: mgrmap e13: y(active, since 6s) 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T21:56:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: ceph telemetry on 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout:For more information see: 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:20.796 INFO:teuthology.orchestra.run.smithi151.stdout:Bootstrap complete. 2024-03-11T21:56:20.838 INFO:tasks.cephadm:Fetching config... 2024-03-11T21:56:20.838 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:56:20.838 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T21:56:21.059 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T21:56:21.059 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:56:21.060 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T21:56:21.115 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T21:56:21.115 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:56:21.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T21:56:21.267 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T21:56:21.267 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:56:21.267 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T21:56:21.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:21 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/2467794009' entity='client.admin' 2024-03-11T21:56:21.322 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T21:56:21.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:56:21.402 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:56:21.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:56:21.463 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:56:21.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:56:21.522 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCWC2a7GBphL2q6eL/i66uOYFszK1D6Etjp2QlhZx7qQQ2V+yid+90JttcZctLUC85Lcu569rymbclOv9QHoF8vUaPoQam2zXp5HThtuPCenkA4HMun/8Eq32SRDgd9UqenyjdlRZH95qe4NvDgA+8aPsu8iwkIhqbrOrVKyGwmjX6GM/Q7t4dWkF4I071mhOJj/rV+AcVhPYHS1XRPGP6KQKEqpAiO/MtyZ2ozHS6K6JSHJWbglc2Kr1b86YdmLMjiPjtX2wdECH/FuxN4tbtBBol6BnJFve4UeKfMkdewGACAcDV+yOi39lcmMLi7jyEBcODwDoIuTZ1jSVtJDG2nEZ9LKueMsbLScocgMxTX+LaJNRkEi0c/Gyrr94wI6Tt9XOqHfOVPhLlRHoy4rAEqyhokLtWOJg9OCMIT8BO0q0m0zZMGe66Uur+N7hrfRJXMridEBvdG0PBH6fsjNH+w+LkbhC9jSvMn8R5QduFyOIktoGK1kw3VGbOWtnh7jJM= ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:56:21.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T21:56:21.841 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:22.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:22.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:22 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:24.309 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T21:56:24.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T21:56:24.797 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:24.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:24 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/3855132011' entity='client.admin' 2024-03-11T21:56:24.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:24.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:26.342 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:26.342 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:28.613 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2024-03-11T21:56:28.614 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:56:28.614 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:56:28.632 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:56:28.632 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:28.690 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2024-03-11T21:56:28.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch host add smithi176 2024-03-11T21:56:29.121 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:29.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:29.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:30.518 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:30 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:30.518 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:30 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:32.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:31 smithi151 ceph-mon[25773]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:32.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:31 smithi151 ceph-mon[25773]: Deploying cephadm binary to smithi176 2024-03-11T21:56:33.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:33 smithi151 ceph-mon[25773]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:34.970 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi176' with addr '172.21.15.176' 2024-03-11T21:56:35.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:56:35.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:35 smithi151 ceph-mon[25773]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:35.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:35 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:35.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:35 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: Added host smithi176 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:36.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:36 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:37.564 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:37.564 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2024-03-11T21:56:37.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi176", "caps": []}]: dispatch 2024-03-11T21:56:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:37 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi176", "caps": []}]': finished 2024-03-11T21:56:38.118 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-03-11T21:56:38.118 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:56:38.119 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:56:38.137 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:56:38.137 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:38.195 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-03-11T21:56:38.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch host add smithi189 2024-03-11T21:56:38.435 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:38.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:38.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: Deploying daemon agent.smithi176 on smithi176 2024-03-11T21:56:38.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:38.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:38.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:38.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:38.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:39.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:39 smithi151 ceph-mon[25773]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:39.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:39 smithi151 ceph-mon[25773]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:39.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:39 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:39.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:39 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:39.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:39 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:41 smithi151 ceph-mon[25773]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:41 smithi151 ceph-mon[25773]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:42.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:42 smithi151 ceph-mon[25773]: Deploying cephadm binary to smithi189 2024-03-11T21:56:42.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:42 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:42.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:42 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:42.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:42 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:43.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:43 smithi151 ceph-mon[25773]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:44.638 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-03-11T21:56:45.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:44 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:45.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:44 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:45.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:44 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:45.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:44 smithi151 ceph-mon[25773]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:45.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:44 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:45.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:44 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:45.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:56:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: Added host smithi189 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:46.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:47.665 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:56:47.665 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-11T21:56:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:47 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]': finished 2024-03-11T21:56:48.305 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T21:56:48.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T21:56:48.539 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:56:48.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: Deploying daemon agent.smithi189 on smithi189 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:49.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:49 smithi151 ceph-mon[25773]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:56:49.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:49 smithi151 ceph-mon[25773]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:49.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:49.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:50.546 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2024-03-11T21:56:50.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:50 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4051278278' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:56:51.145 INFO:tasks.cephadm:Adding mon.a on smithi151 2024-03-11T21:56:51.145 INFO:tasks.cephadm:Adding mon.b on smithi176 2024-03-11T21:56:51.145 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-03-11T21:56:51.145 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi151:172.21.15.151=a;smithi176:172.21.15.176=b;smithi189:172.21.15.189=c' 2024-03-11T21:56:51.429 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:51.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:51 smithi151 ceph-mon[25773]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:51.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:51 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4051278278' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:56:51.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:51 smithi151 ceph-mon[25773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:56:51.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:51 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:51.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:51 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:53.115 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-03-11T21:56:53.720 DEBUG:teuthology.orchestra.run.smithi176:mon.b> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T21:56:53.726 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T21:56:53.728 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:56:53.728 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:56:53.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:53 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:56:54.466 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:56:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:54 smithi151 ceph-mon[25773]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi151:172.21.15.151=a;smithi176:172.21.15.176=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:56:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:54 smithi151 ceph-mon[25773]: Saving service mon spec with placement smithi151:172.21.15.151=a;smithi176:172.21.15.176=b;smithi189:172.21.15.189=c;count:3 2024-03-11T21:56:54.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:54 smithi151 ceph-mon[25773]: Deploying daemon mon.c on smithi189 2024-03-11T21:56:55.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:55 smithi151 ceph-mon[25773]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:57.613 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:56:57.613 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"e42c6b1c-dff1-11ee-95c8-87774f69a715","modified":"2024-03-11T21:55:15.654591Z","created":"2024-03-11T21:55:15.654591Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T21:56:57.618 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-03-11T21:56:57.741 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:56:57 smithi189 ceph-mon[27727]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T21:56:57.741 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:56:57 smithi189 bash[27675]: 69d0055a18d1f9cc66a075c07c2b1063ca8834bfb271a13e922435190e5b7fb3 2024-03-11T21:56:57.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:56:57 smithi151 ceph-mon[25773]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:56:58.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:56:57 smithi189 systemd[1]: Started Ceph mon.c for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:56:59.446 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:56:59.447 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:56:59.692 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:57:03.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:57:03.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: mon.a calling monitor election 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: mon.c calling monitor election 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: monmap e2: 2 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: fsmap 2024-03-11T21:57:03.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: mgrmap e13: y(active, since 50s) 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: overall HEALTH_OK 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:03.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:03.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:57:03.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: mon.a calling monitor election 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: mon.c calling monitor election 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: monmap e2: 2 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: fsmap 2024-03-11T21:57:03.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: mgrmap e13: y(active, since 50s) 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: overall HEALTH_OK 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:03.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:03.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:03 smithi151 ceph-mon[25773]: Deploying daemon mon.b on smithi176 2024-03-11T21:57:03.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:03 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:04.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:03 smithi189 ceph-mon[27727]: Deploying daemon mon.b on smithi176 2024-03-11T21:57:04.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:03 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:04.446 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:57:04.446 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"e42c6b1c-dff1-11ee-95c8-87774f69a715","modified":"2024-03-11T21:56:57.718719Z","created":"2024-03-11T21:55:15.654591Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:57:04.448 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-03-11T21:57:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:05 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:05 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:05 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:05 smithi151 ceph-mon[25773]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:05 smithi151 ceph-mon[25773]: from='client.? 172.21.15.189:0/3949491718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:57:05.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:05 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:05.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:05 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:05.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:05 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:05.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:05 smithi189 ceph-mon[27727]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:05.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:05 smithi189 ceph-mon[27727]: from='client.? 172.21.15.189:0/3949491718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:57:06.020 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:57:06.020 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:57:06.262 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:57:06.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 podman[27584]: 2024-03-11 21:57:06.342144635 +0000 UTC m=+0.017598709 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:07.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 podman[27584]: 2024-03-11 21:57:06.632865924 +0000 UTC m=+0.308319992 container init 3fcc03fa32282a53bacb2c44cff19ae1ed9e2d2d849019eedda50c2ee2967cc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 podman[27584]: 2024-03-11 21:57:06.638475444 +0000 UTC m=+0.313929517 container start 3fcc03fa32282a53bacb2c44cff19ae1ed9e2d2d849019eedda50c2ee2967cc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pidfile_write: ignore empty --pid-file 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: load: jerasure load: lrc 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Git sha 0 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: DB SUMMARY 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: DB Session ID: FMCYR1UNURMXQQZGC9YK 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:57:07.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.env: 0x557915273ae0 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:57:07.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.info_log: 0x557916c318c0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.statistics: (nil) 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.use_fsync: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:57:07.102 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.db_log_dir: 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.wal_dir: 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:57:07.103 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.write_buffer_manager: 0x557916c05400 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.unordered_write: 0 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.row_cache: None 2024-03-11T21:57:07.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.wal_filter: None 2024-03-11T21:57:07.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.wal_compression: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:57:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_open_files: -1 2024-03-11T21:57:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Compression algorithms supported: 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kZSTD supported: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:57:07.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.merge_operator: 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_filter: None 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557916c319c0) 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:57:07.110 INFO:journalctl@ceph.mon.b.smithi176.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: index_type: 0 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: data_block_index_type: 0 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: index_shortening: 1 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: checksum: 4 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: no_block_cache: 0 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache: 0x557916c35090 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache_options: 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: capacity : 536870912 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: num_shard_bits : 4 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: strict_capacity_limit : 0 2024-03-11T21:57:07.111 INFO:journalctl@ceph.mon.b.smithi176.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache_compressed: (nil) 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: persistent_cache: (nil) 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_size: 4096 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_size_deviation: 10 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_restart_interval: 16 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: index_block_restart_interval: 1 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: metadata_block_size: 4096 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: partition_filters: 0 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: use_delta_encoding: 1 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: filter_policy: bloomfilter 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: whole_key_filtering: 1 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: verify_compression: 0 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: format_version: 5 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: enable_index_compression: 1 2024-03-11T21:57:07.112 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_align: 0 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout: max_auto_readahead_size: 262144 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout: prepopulate_block_cache: 0 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression: NoCompression 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.num_levels: 7 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:57:07.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:57:07.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:57:07.115 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:57:07.116 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:57:07.117 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.ttl: 2592000 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:57:07.118 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7fa3461f-7027-4956-b61c-cbe48ab45031 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194226679954, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194226680390, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710194226, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7fa3461f-7027-4956-b61c-cbe48ab45031", "db_session_id": "FMCYR1UNURMXQQZGC9YK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194226680485, "job": 1, "event": "recovery_finished"} 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557916d26000 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: DB pointer 0x557916d10000 2024-03-11T21:57:07.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** DB Stats ** 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:57:07.120 INFO:journalctl@ceph.mon.b.smithi176.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:57:07.121 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: Block cache BinnedLRUCache@0x557916c35090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: using public_addr v2:172.21.15.176:0/0 -> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: starting mon.b rank -1 at public addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] at bind addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:57:07.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(???) e0 preinit fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout: e1 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout: legacy client fscid: -1 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout: No filesystems configured 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:57:07.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Added host smithi176 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:57:07.124 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi176", "caps": []}]: dispatch 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi176", "caps": []}]': finished 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Deploying daemon agent.smithi176 on smithi176 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.125 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Deploying cephadm binary to smithi189 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Added host smithi189 2024-03-11T21:57:07.127 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]': finished 2024-03-11T21:57:07.128 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Deploying daemon agent.smithi189 on smithi189 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.129 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4051278278' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4051278278' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi151:172.21.15.151=a;smithi176:172.21.15.176=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Saving service mon spec with placement smithi151:172.21.15.151=a;smithi176:172.21.15.176=b;smithi189:172.21.15.189=c;count:3 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Deploying daemon mon.c on smithi189 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.a calling monitor election 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.c calling monitor election 2024-03-11T21:57:07.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: monmap e2: 2 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: fsmap 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mgrmap e13: y(active, since 50s) 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: overall HEALTH_OK 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: Deploying daemon mon.b on smithi176 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: from='client.? 172.21.15.189:0/3949491718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 ceph-mon[27622]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 bash[27584]: 3fcc03fa32282a53bacb2c44cff19ae1ed9e2d2d849019eedda50c2ee2967cc9 2024-03-11T21:57:07.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:06 smithi176 systemd[1]: Started Ceph mon.b for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:57:12.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:57:12.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: mon.a calling monitor election 2024-03-11T21:57:12.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: mon.c calling monitor election 2024-03-11T21:57:12.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:12.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:12.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:12.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:57:12.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:12.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: fsmap 2024-03-11T21:57:12.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:12.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: mgrmap e13: y(active, since 59s) 2024-03-11T21:57:12.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:57:12.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:57:12.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:57:12.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-03-11T21:57:12.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:57:12.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: mon.a calling monitor election 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: mon.c calling monitor election 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: fsmap 2024-03-11T21:57:12.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: mgrmap e13: y(active, since 59s) 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:12.388 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-11T21:57:12.388 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"e42c6b1c-dff1-11ee-95c8-87774f69a715","modified":"2024-03-11T21:57:06.703691Z","created":"2024-03-11T21:55:15.654591Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:57:12.390 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-03-11T21:57:13.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:13.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:13.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:13.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:57:13.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:57:13.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='client.? 172.21.15.189:0/1822787961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:13.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:13.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:13.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:13.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:12 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:13.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:13.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='client.? 172.21.15.189:0/1822787961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:13.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:12 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.217 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T21:57:13.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T21:57:13.805 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:57:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:57:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: mon.a calling monitor election 2024-03-11T21:57:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: mon.c calling monitor election 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: fsmap 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:13.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: mgrmap e13: y(active, since 59s) 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:13.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='client.? 172.21.15.189:0/1822787961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.852 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:13.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:13.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:13.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:13.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:13.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:13 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:14.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: mon.b calling monitor election 2024-03-11T21:57:14.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: mon.b calling monitor election 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: mon.a calling monitor election 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: mon.c calling monitor election 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: fsmap 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: mgrmap e13: y(active, since 61s) 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: Cluster is now healthy 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: overall HEALTH_OK 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:14.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: Reconfiguring daemon mon.b on smithi176 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:14.851 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: mon.b calling monitor election 2024-03-11T21:57:15.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: mon.b calling monitor election 2024-03-11T21:57:15.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: mon.a calling monitor election 2024-03-11T21:57:15.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: mon.c calling monitor election 2024-03-11T21:57:15.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:57:15.027 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: fsmap 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: mgrmap e13: y(active, since 61s) 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: Cluster is now healthy 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: overall HEALTH_OK 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: Reconfiguring daemon mon.b on smithi176 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:15.028 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.029 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.029 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: mon.b calling monitor election 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: mon.b calling monitor election 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: mon.a calling monitor election 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: mon.c calling monitor election 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: monmap e3: 3 mons at {a=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: fsmap 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: mgrmap e13: y(active, since 61s) 2024-03-11T21:57:15.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: Cluster is now healthy 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: overall HEALTH_OK 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: Reconfiguring daemon mon.b on smithi176 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:15.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.931 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:15.932 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.932 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.932 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.932 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.932 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:15.932 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.037 INFO:teuthology.orchestra.run.smithi151.stdout:# minimal ceph.conf for e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:57:16.038 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-03-11T21:57:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T21:57:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: mon_host = [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-11T21:57:16.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:57:16.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.624 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T21:57:16.624 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:57:16.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:57:16.654 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:57:16.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:57:16.721 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:57:16.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:57:16.752 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:57:16.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:57:16.819 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:57:16.819 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:57:16.848 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:57:16.848 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/251439911' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: Reconfiguring daemon mon.c on smithi189 2024-03-11T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:16 smithi189 ceph-mon[27727]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:16.915 INFO:tasks.cephadm:Adding mgr.y on smithi151 2024-03-11T21:57:16.916 INFO:tasks.cephadm:Adding mgr.x on smithi176 2024-03-11T21:57:16.916 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-03-11T21:57:16.916 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi151=y;smithi176=x;smithi189=z' 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/251439911' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: Reconfiguring daemon mon.c on smithi189 2024-03-11T21:57:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:16 smithi151 ceph-mon[25773]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/251439911' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: Reconfiguring daemon mon.c on smithi189 2024-03-11T21:57:17.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:16 smithi176 ceph-mon[27622]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:17.866 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:57:19.046 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.046 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.046 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:19.046 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:19.047 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:19.047 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.047 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:18 smithi189 ceph-mon[27727]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:19.098 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.098 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.098 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:19.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:19.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:19.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:18 smithi176 ceph-mon[27622]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:19.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:18 smithi151 ceph-mon[25773]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:19.630 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-03-11T21:57:20.214 DEBUG:teuthology.orchestra.run.smithi176:mgr.x> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T21:57:20.217 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T21:57:20.219 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T21:57:20.219 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:57:20.220 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:57:20.239 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:57:20.239 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme1n1 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme1n1 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:57:03.623128497 +0000 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:54:06.288463778 +0000 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:54:06.288463778 +0000 2024-03-11T21:57:20.297 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:54:06.279463897 +0000 2024-03-11T21:57:20.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:57:20.367 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:57:20.367 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:57:20.367 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000308308 s, 1.7 MB/s 2024-03-11T21:57:20.368 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:57:20.427 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme2n1 2024-03-11T21:57:20.484 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme2n1 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:57:03.635128339 +0000 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:54:06.445461710 +0000 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:54:06.445461710 +0000 2024-03-11T21:57:20.485 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:54:06.436461828 +0000 2024-03-11T21:57:20.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:57:20.553 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:57:20.553 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:57:20.553 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000289773 s, 1.8 MB/s 2024-03-11T21:57:20.554 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:57:20.612 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme3n1 2024-03-11T21:57:20.669 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme3n1 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:57:03.646128194 +0000 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:54:06.603459630 +0000 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:54:06.603459630 +0000 2024-03-11T21:57:20.670 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:54:06.595459735 +0000 2024-03-11T21:57:20.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:57:20.737 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:57:20.737 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:57:20.737 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000239763 s, 2.1 MB/s 2024-03-11T21:57:20.738 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:57:20.795 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme4n1 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme4n1 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-11 21:57:03.658128036 +0000 2024-03-11T21:57:20.851 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-11 21:54:06.758457588 +0000 2024-03-11T21:57:20.852 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-11 21:54:06.758457588 +0000 2024-03-11T21:57:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-11 21:54:06.747457733 +0000 2024-03-11T21:57:20.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:57:20.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi151=y;smithi176=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:20.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: Saving service mgr spec with placement smithi151=y;smithi176=x;smithi189=z;count:3 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: Deploying daemon mgr.z on smithi189 2024-03-11T21:57:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:20 smithi151 ceph-mon[25773]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:20.918 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-11T21:57:20.918 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-11T21:57:20.918 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000287025 s, 1.8 MB/s 2024-03-11T21:57:20.920 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:57:20.976 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:57:20.976 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:57:20.995 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:57:20.996 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme1n1 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi151=y;smithi176=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: Saving service mgr spec with placement smithi151=y;smithi176=x;smithi189=z;count:3 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:21.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:57:21.033 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:21.033 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:21.033 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: Deploying daemon mgr.z on smithi189 2024-03-11T21:57:21.033 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:20 smithi189 ceph-mon[27727]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:21.053 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme1n1 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:57:14.179681356 +0000 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:54:07.958504429 +0000 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:54:07.958504429 +0000 2024-03-11T21:57:21.054 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:54:07.946504547 +0000 2024-03-11T21:57:21.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:57:21.098 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi151=y;smithi176=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: Saving service mgr spec with placement smithi151=y;smithi176=x;smithi189=z;count:3 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:57:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:21.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:21.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: Deploying daemon mgr.z on smithi189 2024-03-11T21:57:21.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:20 smithi176 ceph-mon[27622]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:21.126 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:57:21.127 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:57:21.127 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000303429 s, 1.7 MB/s 2024-03-11T21:57:21.127 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:57:21.184 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme2n1 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme2n1 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:57:14.191681239 +0000 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:54:08.106502978 +0000 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:54:08.106502978 +0000 2024-03-11T21:57:21.240 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:54:08.095503086 +0000 2024-03-11T21:57:21.240 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:57:21.307 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:57:21.307 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:57:21.307 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000269528 s, 1.9 MB/s 2024-03-11T21:57:21.308 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:57:21.365 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme3n1 2024-03-11T21:57:21.421 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme3n1 2024-03-11T21:57:21.421 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:21.421 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T21:57:21.421 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:21.421 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:21.422 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:57:14.201681141 +0000 2024-03-11T21:57:21.422 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:54:08.253501536 +0000 2024-03-11T21:57:21.422 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:54:08.253501536 +0000 2024-03-11T21:57:21.422 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:54:08.241501654 +0000 2024-03-11T21:57:21.422 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:57:21.488 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:57:21.488 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:57:21.488 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000213701 s, 2.4 MB/s 2024-03-11T21:57:21.489 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:57:21.546 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme4n1 2024-03-11T21:57:21.603 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme4n1 2024-03-11T21:57:21.603 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:21.603 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:57:21.603 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:21.603 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:21.604 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-11 21:57:14.212681034 +0000 2024-03-11T21:57:21.604 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-11 21:54:08.400500094 +0000 2024-03-11T21:57:21.604 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-11 21:54:08.400500094 +0000 2024-03-11T21:57:21.604 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-11 21:54:08.389500202 +0000 2024-03-11T21:57:21.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:57:21.669 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-11T21:57:21.669 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-11T21:57:21.669 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000288894 s, 1.8 MB/s 2024-03-11T21:57:21.670 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:57:21.727 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:57:21.727 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:57:21.743 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:57:21.744 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme1n1 2024-03-11T21:57:21.801 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme1n1 2024-03-11T21:57:21.801 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:21.801 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:57:21.802 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:21.802 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:21.802 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:57:14.695122892 +0000 2024-03-11T21:57:21.802 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:54:09.607897237 +0000 2024-03-11T21:57:21.802 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:54:09.607897237 +0000 2024-03-11T21:57:21.802 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:54:09.595897484 +0000 2024-03-11T21:57:21.802 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:57:21.868 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:21 smithi189 systemd[1]: Starting Ceph mgr.z for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:57:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:57:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000225932 s, 2.3 MB/s 2024-03-11T21:57:21.874 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:57:21.930 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme2n1 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme2n1 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:57:14.706122671 +0000 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:54:09.761894064 +0000 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:54:09.761894064 +0000 2024-03-11T21:57:21.988 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:54:09.750894291 +0000 2024-03-11T21:57:21.989 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:57:22.055 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:57:22.055 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:57:22.056 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000222679 s, 2.3 MB/s 2024-03-11T21:57:22.057 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:57:22.114 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme3n1 2024-03-11T21:57:22.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:21 smithi189 podman[30055]: 2024-03-11 21:57:21.858541327 +0000 UTC m=+0.018960461 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:22.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:21 smithi189 podman[30055]: 2024-03-11 21:57:21.974768798 +0000 UTC m=+0.135187960 container create 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T21:57:22.142 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme3n1 2024-03-11T21:57:22.142 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:22.142 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T21:57:22.142 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:22.142 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:22.143 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:57:14.717122449 +0000 2024-03-11T21:57:22.143 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:54:09.913890933 +0000 2024-03-11T21:57:22.143 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:54:09.913890933 +0000 2024-03-11T21:57:22.143 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:54:09.901891180 +0000 2024-03-11T21:57:22.143 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:57:22.210 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:57:22.211 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:57:22.211 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000237225 s, 2.2 MB/s 2024-03-11T21:57:22.212 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:57:22.269 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme4n1 2024-03-11T21:57:22.324 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme4n1 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-11 21:57:14.728122227 +0000 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-11 21:54:10.064887822 +0000 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-11 21:54:10.064887822 +0000 2024-03-11T21:57:22.325 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-11 21:54:10.052888069 +0000 2024-03-11T21:57:22.325 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:57:22.393 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-11T21:57:22.393 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-11T21:57:22.393 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000240297 s, 2.1 MB/s 2024-03-11T21:57:22.394 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:57:22.450 INFO:tasks.cephadm:Deploying osd.0 on smithi151 with /dev/nvme4n1... 2024-03-11T21:57:22.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:57:22.480 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 podman[30055]: 2024-03-11 21:57:22.210630897 +0000 UTC m=+0.371050018 container init 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T21:57:22.480 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 podman[30055]: 2024-03-11 21:57:22.216092798 +0000 UTC m=+0.376511919 container start 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:57:22.480 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 ceph-mgr[30216]: -- 172.21.15.189:0/3652993953 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d44dd641a0 con 0x55d44dd3ac00 2024-03-11T21:57:22.480 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 bash[30055]: 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 2024-03-11T21:57:22.480 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 systemd[1]: Started Ceph mgr.z for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:57:22.879 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:57:23.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:22.783+0000 7faf2482a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:57:23.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:22 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:22.877+0000 7faf2482a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:23.518 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:23.519 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-mon[27727]: Deploying daemon mgr.x on smithi176 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:23 smithi151 ceph-mon[25773]: Deploying daemon mgr.x on smithi176 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:57:23.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:23.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:23.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:23 smithi176 ceph-mon[27622]: Deploying daemon mgr.x on smithi176 2024-03-11T21:57:23.845 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:23.523+0000 7faf2482a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:57:23.845 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:23.616+0000 7faf2482a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:57:23.845 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:23.755+0000 7faf2482a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:57:24.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:23.850+0000 7faf2482a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:57:24.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:23 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:23.932+0000 7faf2482a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:57:24.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:24 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:24.202+0000 7faf2482a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:57:24.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:24 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:24.310+0000 7faf2482a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:57:24.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:24 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:24.395+0000 7faf2482a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:57:24.486 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:24 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:24.488+0000 7faf2482a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:57:24.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 systemd[1]: Starting Ceph mgr.x for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:57:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:57:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:57:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:57:24.848 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-11T21:57:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-11T21:57:24.848 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145486 s, 721 MB/s 2024-03-11T21:57:24.849 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-11T21:57:24.877 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:24 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:24.663+0000 7faf2482a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:57:24.925 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 podman[29301]: 2024-03-11 21:57:24.562828741 +0000 UTC m=+0.017687844 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:57:24.925 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 podman[29301]: 2024-03-11 21:57:24.662867769 +0000 UTC m=+0.117726863 container create c86fd27ac212532b555187886f5c8b22d4f7afe82b242199ecdfdf2e6a04d2e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T21:57:24.925 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 podman[29301]: 2024-03-11 21:57:24.865491238 +0000 UTC m=+0.320350332 container init c86fd27ac212532b555187886f5c8b22d4f7afe82b242199ecdfdf2e6a04d2e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-11T21:57:24.925 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 podman[29301]: 2024-03-11 21:57:24.87086067 +0000 UTC m=+0.325719767 container start c86fd27ac212532b555187886f5c8b22d4f7afe82b242199ecdfdf2e6a04d2e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:57:24.926 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 ceph-mgr[29339]: -- 172.21.15.176:0/3937322812 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a9d2c141a0 con 0x55a9d2beb000 2024-03-11T21:57:24.926 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 bash[29301]: c86fd27ac212532b555187886f5c8b22d4f7afe82b242199ecdfdf2e6a04d2e0 2024-03-11T21:57:25.336 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:24 smithi176 systemd[1]: Started Ceph mgr.x for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:25 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi151:/dev/nvme4n1 2024-03-11T21:57:25.598 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:25.442+0000 7eff9293d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:57:25.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:25 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:25.540+0000 7eff9293d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:25.737 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:57:25.896 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:25.634+0000 7faf2482a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:57:25.896 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:25 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:25.761+0000 7faf2482a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:57:26.148 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:26.111+0000 7faf2482a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:26.237 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-mon[27622]: Reconfiguring daemon mgr.y on smithi151 2024-03-11T21:57:26.401 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:26.217+0000 7faf2482a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:26.402 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:26.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:26.403 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-mon[27727]: Reconfiguring daemon mgr.y on smithi151 2024-03-11T21:57:26.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:26.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:26.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:26.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:26 smithi151 ceph-mon[25773]: Reconfiguring daemon mgr.y on smithi151 2024-03-11T21:57:26.522 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:26.246+0000 7eff9293d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:57:26.522 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:26.344+0000 7eff9293d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:57:26.522 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:26.489+0000 7eff9293d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:57:26.818 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:26.415+0000 7faf2482a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:57:26.818 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:26.613+0000 7faf2482a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:57:26.848 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:26.590+0000 7eff9293d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:57:26.848 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:26.680+0000 7eff9293d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:57:27.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:26 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:26.899+0000 7faf2482a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:57:27.156 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:27 smithi176 ceph-mon[27622]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:27.156 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:26.950+0000 7eff9293d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:57:27.156 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:27.073+0000 7eff9293d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:57:27.157 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:27.162+0000 7eff9293d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:57:27.434 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:27 smithi189 ceph-mon[27727]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:27.434 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:27 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:27.174+0000 7faf2482a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:57:27.434 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:27 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:27.438+0000 7faf2482a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:57:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:27 smithi151 ceph-mon[25773]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:27.454 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:27.268+0000 7eff9293d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:57:27.454 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:27.459+0000 7eff9293d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:57:28.463 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:28.317+0000 7faf2482a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:57:28.540 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.541 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:28.515+0000 7eff9293d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:57:28.743 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.743 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.744 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.744 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:28.744 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:28.744 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:28.744 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:28.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:28.848 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:28 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:28.651+0000 7eff9293d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:57:29.005 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:28 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:28.906+0000 7faf2482a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:57:29.005 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:29.010+0000 7faf2482a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:57:29.278 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:28.999+0000 7eff9293d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:57:29.278 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:29.088+0000 7eff9293d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:57:29.278 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:29.284+0000 7eff9293d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:57:29.377 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:57:29.227+0000 7faf2482a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:57:29.598 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:29.468+0000 7eff9293d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: Standby manager daemon z started 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:57:29.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:29.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: Standby manager daemon z started 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:57:29.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:57:29.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:57:29.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:29 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:57:29.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:57:29.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:29.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:29.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:29.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:29.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: Standby manager daemon z started 2024-03-11T21:57:29.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:57:29.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:57:29.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:57:29.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:29 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/3267212815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:57:30.007 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:29.752+0000 7eff9293d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:57:30.007 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:30 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:30.012+0000 7eff9293d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:57:30.348 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:30 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:30.245+0000 7eff9293d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:57:31.349 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:31.034+0000 7eff9293d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:57:31.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-mon[27622]: mgrmap e14: y(active, since 77s), standbys: z 2024-03-11T21:57:31.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:57:31.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-mon[27622]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:31.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:31 smithi189 ceph-mon[27727]: mgrmap e14: y(active, since 77s), standbys: z 2024-03-11T21:57:31.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:31 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:57:31.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:31 smithi189 ceph-mon[27727]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:31.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:31 smithi151 ceph-mon[25773]: mgrmap e14: y(active, since 77s), standbys: z 2024-03-11T21:57:31.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:31 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:57:31.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:31 smithi151 ceph-mon[25773]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:31.907 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:31.605+0000 7eff9293d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:57:31.907 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:31.701+0000 7eff9293d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:57:31.907 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:57:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:57:31.913+0000 7eff9293d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/200426170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f5c84c-f5c7-4613-aedc-d69504d53a56"}]: dispatch 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/200426170' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71f5c84c-f5c7-4613-aedc-d69504d53a56"}]': finished 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: Standby manager daemon x started 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:57:32.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:32 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/200426170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f5c84c-f5c7-4613-aedc-d69504d53a56"}]: dispatch 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/200426170' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71f5c84c-f5c7-4613-aedc-d69504d53a56"}]': finished 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: Standby manager daemon x started 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:57:32.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:32 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/200426170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f5c84c-f5c7-4613-aedc-d69504d53a56"}]: dispatch 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/200426170' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71f5c84c-f5c7-4613-aedc-d69504d53a56"}]': finished 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: Standby manager daemon x started 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:57:32.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:32 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/587914453' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:57:33.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:33 smithi151 ceph-mon[25773]: mgrmap e15: y(active, since 79s), standbys: z, x 2024-03-11T21:57:33.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:33 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:57:33.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:33 smithi151 ceph-mon[25773]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:33.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:33 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/349046613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:57:33.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:33 smithi176 ceph-mon[27622]: mgrmap e15: y(active, since 79s), standbys: z, x 2024-03-11T21:57:33.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:33 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:57:33.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:33 smithi176 ceph-mon[27622]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:33.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:33 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/349046613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:57:33.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:33 smithi189 ceph-mon[27727]: mgrmap e15: y(active, since 79s), standbys: z, x 2024-03-11T21:57:33.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:33 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:57:33.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:33 smithi189 ceph-mon[27727]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:33.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:33 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/349046613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:57:35.754 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:35 smithi176 ceph-mon[27622]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:35.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:35 smithi151 ceph-mon[25773]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:35.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:35 smithi189 ceph-mon[27727]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:37 smithi151 ceph-mon[25773]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:37.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:37 smithi176 ceph-mon[27622]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:37.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:37 smithi189 ceph-mon[27727]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:38.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:57:38.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:38.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:57:38.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:38.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:57:38.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:39.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:39 smithi151 ceph-mon[25773]: Deploying daemon osd.0 on smithi151 2024-03-11T21:57:39.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:39 smithi151 ceph-mon[25773]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:39.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:39 smithi176 ceph-mon[27622]: Deploying daemon osd.0 on smithi151 2024-03-11T21:57:39.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:39 smithi176 ceph-mon[27622]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:39.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:39 smithi189 ceph-mon[27727]: Deploying daemon osd.0 on smithi151 2024-03-11T21:57:39.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:39 smithi189 ceph-mon[27727]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:41.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:41 smithi151 ceph-mon[25773]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:41 smithi176 ceph-mon[27622]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:41.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:41 smithi189 ceph-mon[27727]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:43.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:43 smithi151 ceph-mon[25773]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:43.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:43 smithi176 ceph-mon[27622]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:43.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:43 smithi189 ceph-mon[27727]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:45.215 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:45.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:45.217 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:45.217 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.217 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:45 smithi151 ceph-mon[25773]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:45.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:45.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:45.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:45 smithi176 ceph-mon[27622]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:45.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:45.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:45.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:45.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.379 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.379 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.379 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.379 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:45.379 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:45.379 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:45.380 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:45.380 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:45 smithi189 ceph-mon[27727]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:46.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.733 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:46 smithi151 ceph-mon[25773]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:46.862 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.863 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:46 smithi189 ceph-mon[27727]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:46.867 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:46 smithi176 ceph-mon[27622]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:47.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:47 smithi151 ceph-mon[25773]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:57:47.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:47 smithi189 ceph-mon[27727]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:57:47.931 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:47 smithi176 ceph-mon[27622]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:57:48.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:57:48.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:57:48.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-11T21:57:48.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:48.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:48 smithi151 ceph-mon[25773]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:48 smithi176 ceph-mon[27622]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:48.953 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:57:48.954 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:57:48.954 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-11T21:57:48.954 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:48.954 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.954 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.954 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.955 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.955 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.955 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:48.955 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:48 smithi189 ceph-mon[27727]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:49.072 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 0 on host 'smithi151' 2024-03-11T21:57:49.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-11T21:57:49.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:57:49.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:49.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:49.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:49.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:57:49.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:49.746 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: Adjusting osd_memory_target on smithi151 to 16064M 2024-03-11T21:57:49.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:49.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:49.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:49.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:49.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:49.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' 2024-03-11T21:57:49.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:49.774 DEBUG:teuthology.orchestra.run.smithi151:osd.0> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T21:57:49.777 INFO:tasks.cephadm:Deploying osd.1 on smithi176 with /dev/nvme4n1... 2024-03-11T21:57:49.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:57:50.012 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:57:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-11T21:57:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:57:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: Adjusting osd_memory_target on smithi151 to 16064M 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' 2024-03-11T21:57:50.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:50.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-11T21:57:50.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: Adjusting osd_memory_target on smithi151 to 16064M 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461]' entity='osd.0' 2024-03-11T21:57:50.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:51.030 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:50 smithi189 ceph-mon[27727]: purged_snaps scrub starts 2024-03-11T21:57:51.030 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:50 smithi189 ceph-mon[27727]: purged_snaps scrub ok 2024-03-11T21:57:51.030 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:50 smithi189 ceph-mon[27727]: osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461] boot 2024-03-11T21:57:51.030 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:50 smithi189 ceph-mon[27727]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:57:51.030 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:50 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:51.030 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:50 smithi189 ceph-mon[27727]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:51.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:50 smithi151 ceph-mon[25773]: purged_snaps scrub starts 2024-03-11T21:57:51.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:50 smithi151 ceph-mon[25773]: purged_snaps scrub ok 2024-03-11T21:57:51.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:50 smithi151 ceph-mon[25773]: osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461] boot 2024-03-11T21:57:51.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:50 smithi151 ceph-mon[25773]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:57:51.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:50 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:51.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:50 smithi151 ceph-mon[25773]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:51.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:50 smithi176 ceph-mon[27622]: purged_snaps scrub starts 2024-03-11T21:57:51.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:50 smithi176 ceph-mon[27622]: purged_snaps scrub ok 2024-03-11T21:57:51.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:50 smithi176 ceph-mon[27622]: osd.0 [v2:172.21.15.151:6802/265400461,v1:172.21.15.151:6803/265400461] boot 2024-03-11T21:57:51.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:50 smithi176 ceph-mon[27622]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:57:51.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:50 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:57:51.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:50 smithi176 ceph-mon[27622]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132337 s, 792 MB/s 2024-03-11T21:57:51.929 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-11T21:57:52.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi176:/dev/nvme4n1 2024-03-11T21:57:52.749 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:57:52.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:52 smithi176 ceph-mon[27622]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:57:52.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:52 smithi176 ceph-mon[27622]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:53.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:52 smithi151 ceph-mon[25773]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:57:53.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:52 smithi151 ceph-mon[25773]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:53.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:52 smithi189 ceph-mon[27727]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:57:53.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:52 smithi189 ceph-mon[27727]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:55.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:55 smithi176 ceph-mon[27622]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:55.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:55 smithi176 ceph-mon[27622]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:55.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:55 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:57:55.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:55 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:57:55.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:55 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:55.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:55 smithi151 ceph-mon[25773]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:55.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:55 smithi151 ceph-mon[25773]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:55.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:55 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:57:55.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:55 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:57:55.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:55 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:55.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:55 smithi189 ceph-mon[27727]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:55.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:55 smithi189 ceph-mon[27727]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:57:55.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:55 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:57:55.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:55 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:57:55.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:55 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:57:57.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:57 smithi151 ceph-mon[25773]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:57.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:57 smithi151 ceph-mon[25773]: from='client.? 172.21.15.176:0/502902609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]: dispatch 2024-03-11T21:57:57.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:57 smithi151 ceph-mon[25773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]: dispatch 2024-03-11T21:57:57.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:57 smithi151 ceph-mon[25773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]': finished 2024-03-11T21:57:57.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:57 smithi151 ceph-mon[25773]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:57:57.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:57:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:57 smithi176 ceph-mon[27622]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:57 smithi176 ceph-mon[27622]: from='client.? 172.21.15.176:0/502902609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]: dispatch 2024-03-11T21:57:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:57 smithi176 ceph-mon[27622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]: dispatch 2024-03-11T21:57:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:57 smithi176 ceph-mon[27622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]': finished 2024-03-11T21:57:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:57 smithi176 ceph-mon[27622]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:57:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:57:57.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:57 smithi189 ceph-mon[27727]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:57.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:57 smithi189 ceph-mon[27727]: from='client.? 172.21.15.176:0/502902609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]: dispatch 2024-03-11T21:57:57.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:57 smithi189 ceph-mon[27727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]: dispatch 2024-03-11T21:57:57.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:57 smithi189 ceph-mon[27727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53b2450f-2382-4169-a14d-8ad8087a88f7"}]': finished 2024-03-11T21:57:57.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:57 smithi189 ceph-mon[27727]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:57:57.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:57:58.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:58 smithi151 ceph-mon[25773]: from='client.? 172.21.15.176:0/4090686619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:57:58.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:58 smithi176 ceph-mon[27622]: from='client.? 172.21.15.176:0/4090686619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:57:58.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:58 smithi189 ceph-mon[27727]: from='client.? 172.21.15.176:0/4090686619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:57:59.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:57:59 smithi151 ceph-mon[25773]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:57:59 smithi176 ceph-mon[27622]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:57:59.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:57:59 smithi189 ceph-mon[27727]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:01.755 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:01 smithi176 ceph-mon[27622]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:01.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:01 smithi151 ceph-mon[25773]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:01.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:01 smithi189 ceph-mon[27727]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:03.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:03 smithi176 ceph-mon[27622]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:03.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:03 smithi151 ceph-mon[25773]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:03.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:03 smithi189 ceph-mon[27727]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:04.704 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:04 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:58:04.704 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:04 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:04.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:04 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:58:04.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:04 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:04.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:04 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:58:04.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:04 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:05.591 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:05 smithi176 ceph-mon[27622]: Deploying daemon osd.1 on smithi176 2024-03-11T21:58:05.591 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:05 smithi176 ceph-mon[27622]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:05.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:05 smithi151 ceph-mon[25773]: Deploying daemon osd.1 on smithi176 2024-03-11T21:58:05.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:05 smithi151 ceph-mon[25773]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:05.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:05 smithi189 ceph-mon[27727]: Deploying daemon osd.1 on smithi176 2024-03-11T21:58:05.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:05 smithi189 ceph-mon[27727]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:06.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:06 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:06.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:06 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:06.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:06 smithi151 ceph-mon[25773]: Detected new or changed devices on smithi151 2024-03-11T21:58:06.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:06 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:06.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:06 smithi151 ceph-mon[25773]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:06.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:06.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:06.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:06 smithi176 ceph-mon[27622]: Detected new or changed devices on smithi151 2024-03-11T21:58:06.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:06 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:06.848 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:06 smithi176 ceph-mon[27622]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:07.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:06 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:07.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:06 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:07.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:06 smithi189 ceph-mon[27727]: Detected new or changed devices on smithi151 2024-03-11T21:58:07.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:06 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:07.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:06 smithi189 ceph-mon[27727]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:09.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:08 smithi189 ceph-mon[27727]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:08 smithi151 ceph-mon[25773]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:08 smithi176 ceph-mon[27622]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.487 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.487 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.487 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.487 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.487 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:10.792 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:10.792 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:10.792 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.792 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.792 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.793 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:58:10.793 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:10 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:58:10.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:10 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:10.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:58:10.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:10 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:11.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:11 smithi176 ceph-mon[27622]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T21:58:11.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:11 smithi176 ceph-mon[27622]: Reconfiguring daemon osd.1 on smithi176 2024-03-11T21:58:11.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:11 smithi176 ceph-mon[27622]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T21:58:11.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:11 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:11.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:11 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:11.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:11 smithi151 ceph-mon[25773]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T21:58:11.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:11 smithi151 ceph-mon[25773]: Reconfiguring daemon osd.1 on smithi176 2024-03-11T21:58:11.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:11 smithi151 ceph-mon[25773]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T21:58:11.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:11.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:11 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:11.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:11 smithi189 ceph-mon[27727]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T21:58:11.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:11 smithi189 ceph-mon[27727]: Reconfiguring daemon osd.1 on smithi176 2024-03-11T21:58:11.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:11 smithi189 ceph-mon[27727]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T21:58:11.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:11.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:11 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:12.719 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:12 smithi176 ceph-mon[27622]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:12.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:12 smithi151 ceph-mon[25773]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:12.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:12 smithi189 ceph-mon[27727]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:13.657 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:13 smithi176 ceph-mon[27622]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:13.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:13 smithi189 ceph-mon[27727]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:14.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:13 smithi151 ceph-mon[25773]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:14.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:58:14.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:14.831 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:14 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:58:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:58:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:14.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:14.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:14.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:14.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:14 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:15.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:58:15.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:58:15.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:15.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:15.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:15.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:15.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:15.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:15.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:14 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: from='osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T21:58:15.817 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:15 smithi176 ceph-mon[27622]: Cluster is now healthy 2024-03-11T21:58:15.929 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 1 on host 'smithi176' 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: from='osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T21:58:16.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:15 smithi151 ceph-mon[25773]: Cluster is now healthy 2024-03-11T21:58:16.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:58:16.129 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:58:16.130 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: from='osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-11T21:58:16.130 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:16.130 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-11T21:58:16.130 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T21:58:16.130 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:15 smithi189 ceph-mon[27727]: Cluster is now healthy 2024-03-11T21:58:16.679 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-11T21:58:16.679 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:58:16.679 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:16.679 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:16.679 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:16.679 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: Adjusting osd_memory_target on smithi176 to 16064M 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: from='osd.1 ' entity='osd.1' 2024-03-11T21:58:16.680 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:16 smithi176 ceph-mon[27622]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:16.706 DEBUG:teuthology.orchestra.run.smithi176:osd.1> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:58:16.708 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/nvme4n1... 2024-03-11T21:58:16.708 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:58:16.958 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: Adjusting osd_memory_target on smithi176 to 16064M 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: from='osd.1 ' entity='osd.1' 2024-03-11T21:58:17.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:16 smithi151 ceph-mon[25773]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:58:17.064 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: Adjusting osd_memory_target on smithi176 to 16064M 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: from='osd.1 ' entity='osd.1' 2024-03-11T21:58:17.065 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:16 smithi189 ceph-mon[27727]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:58:18.010 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:17 smithi189 ceph-mon[27727]: purged_snaps scrub starts 2024-03-11T21:58:18.010 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:17 smithi189 ceph-mon[27727]: purged_snaps scrub ok 2024-03-11T21:58:18.010 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:17 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:18.011 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:17 smithi189 ceph-mon[27727]: osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516] boot 2024-03-11T21:58:18.011 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:17 smithi189 ceph-mon[27727]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:58:18.011 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:17 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:18.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:17 smithi151 ceph-mon[25773]: purged_snaps scrub starts 2024-03-11T21:58:18.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:17 smithi151 ceph-mon[25773]: purged_snaps scrub ok 2024-03-11T21:58:18.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:17 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:18.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:17 smithi151 ceph-mon[25773]: osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516] boot 2024-03-11T21:58:18.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:17 smithi151 ceph-mon[25773]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:58:18.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:17 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:18.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:17 smithi176 ceph-mon[27622]: purged_snaps scrub starts 2024-03-11T21:58:18.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:17 smithi176 ceph-mon[27622]: purged_snaps scrub ok 2024-03-11T21:58:18.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:17 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:18.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:17 smithi176 ceph-mon[27622]: osd.1 [v2:172.21.15.176:6800/1458376516,v1:172.21.15.176:6801/1458376516] boot 2024-03-11T21:58:18.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:17 smithi176 ceph-mon[27622]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:58:18.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:17 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123876 s, 846 MB/s 2024-03-11T21:58:18.833 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-11T21:58:19.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:18 smithi151 ceph-mon[25773]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:19.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:18 smithi176 ceph-mon[27622]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:19.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:18 smithi189 ceph-mon[27727]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:19.419 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi189:/dev/nvme4n1 2024-03-11T21:58:19.659 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:58:19.781 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:19 smithi189 ceph-mon[27727]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:58:20.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:19 smithi151 ceph-mon[25773]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:58:20.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:19 smithi176 ceph-mon[27622]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:58:20.910 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:20 smithi189 ceph-mon[27727]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:21.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:20 smithi151 ceph-mon[25773]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:21.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:20 smithi176 ceph-mon[27622]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:21.643 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:21 smithi189 ceph-mon[27727]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:58:21.643 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:21 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:58:21.643 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:21 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:58:21.942 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:21 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:22.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:21 smithi151 ceph-mon[25773]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:58:22.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:21 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:58:22.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:21 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:58:22.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:21 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:22.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:21 smithi176 ceph-mon[27622]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:58:22.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:21 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:58:22.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:21 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:58:22.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:21 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:22.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:22 smithi189 ceph-mon[27727]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:23.000 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:22 smithi176 ceph-mon[27622]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:23.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:22 smithi151 ceph-mon[25773]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:24.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:24 smithi151 ceph-mon[25773]: from='client.? 172.21.15.189:0/3961650107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]: dispatch 2024-03-11T21:58:24.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:24 smithi151 ceph-mon[25773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]: dispatch 2024-03-11T21:58:24.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:24 smithi151 ceph-mon[25773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]': finished 2024-03-11T21:58:24.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:24 smithi151 ceph-mon[25773]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:58:24.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:24.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:24 smithi176 ceph-mon[27622]: from='client.? 172.21.15.189:0/3961650107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]: dispatch 2024-03-11T21:58:24.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:24 smithi176 ceph-mon[27622]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]: dispatch 2024-03-11T21:58:24.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:24 smithi176 ceph-mon[27622]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]': finished 2024-03-11T21:58:24.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:24 smithi176 ceph-mon[27622]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:58:24.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:24.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:24 smithi189 ceph-mon[27727]: from='client.? 172.21.15.189:0/3961650107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]: dispatch 2024-03-11T21:58:24.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:24 smithi189 ceph-mon[27727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]: dispatch 2024-03-11T21:58:24.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:24 smithi189 ceph-mon[27727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0393c20b-b0df-49c3-b073-e73fd05e5c09"}]': finished 2024-03-11T21:58:24.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:24 smithi189 ceph-mon[27727]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:58:24.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:25.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:25 smithi151 ceph-mon[25773]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:25.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:25 smithi176 ceph-mon[27622]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:25.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:25 smithi189 ceph-mon[27727]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:26.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:26 smithi151 ceph-mon[25773]: from='client.? 172.21.15.189:0/3505557472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:58:26.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:26 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:26 smithi176 ceph-mon[27622]: from='client.? 172.21.15.189:0/3505557472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:58:26.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:26 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:26 smithi189 ceph-mon[27727]: from='client.? 172.21.15.189:0/3505557472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:58:26.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:26.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:26 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:27.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:27 smithi176 ceph-mon[27622]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:27.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:27 smithi189 ceph-mon[27727]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:27.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:27 smithi151 ceph-mon[25773]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:29.457 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.457 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.457 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.457 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:29.457 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.457 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.458 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:29 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:29 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:29.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:29 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:30.481 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:30 smithi176 ceph-mon[27622]: Detected new or changed devices on smithi176 2024-03-11T21:58:30.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:30 smithi189 ceph-mon[27727]: Detected new or changed devices on smithi176 2024-03-11T21:58:30.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:30 smithi151 ceph-mon[25773]: Detected new or changed devices on smithi176 2024-03-11T21:58:31.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:31 smithi189 ceph-mon[27727]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:31.731 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:31 smithi176 ceph-mon[27622]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:31.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:31 smithi151 ceph-mon[25773]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:32.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:32 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:58:32.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:32 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:32.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:32 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:58:32.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:32 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:32.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:32 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:58:32.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:32 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:33.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:33 smithi189 ceph-mon[27727]: Deploying daemon osd.2 on smithi189 2024-03-11T21:58:33.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:33 smithi189 ceph-mon[27727]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:33.655 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:33 smithi176 ceph-mon[27622]: Deploying daemon osd.2 on smithi189 2024-03-11T21:58:33.655 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:33 smithi176 ceph-mon[27622]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:33.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:33 smithi151 ceph-mon[25773]: Deploying daemon osd.2 on smithi189 2024-03-11T21:58:33.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:33 smithi151 ceph-mon[25773]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:35.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:35 smithi189 ceph-mon[27727]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:35.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:35 smithi151 ceph-mon[25773]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:35.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:35 smithi176 ceph-mon[27622]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:37.743 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:37 smithi189 ceph-mon[27727]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:37.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:37 smithi151 ceph-mon[25773]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:37.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:37 smithi176 ceph-mon[27622]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:39.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:39.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:39.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:39.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:38 smithi151 ceph-mon[25773]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:39.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:39.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:39.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:39.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:38 smithi176 ceph-mon[27622]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:39.107 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:38 smithi189 ceph-mon[27727]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:40.962 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-03-11T21:58:41.508 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:41.508 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.508 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:41.509 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:41 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.535 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T21:58:41.538 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-11T21:58:41.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:58:41.794 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:41.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:41 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:41.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:58:41.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:41.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:41.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:41.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:41 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:42.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:42 smithi151 ceph-mon[25773]: Adjusting osd_memory_target on smithi189 to 16064M 2024-03-11T21:58:42.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:42 smithi151 ceph-mon[25773]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:58:42.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:42 smithi151 ceph-mon[25773]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:58:42.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:42 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:42.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:42 smithi151 ceph-mon[25773]: from='osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-11T21:58:42.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:42 smithi151 ceph-mon[25773]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-11T21:58:42.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:42 smithi176 ceph-mon[27622]: Adjusting osd_memory_target on smithi189 to 16064M 2024-03-11T21:58:42.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:42 smithi176 ceph-mon[27622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:58:42.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:42 smithi176 ceph-mon[27622]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:58:42.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:42 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:42.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:42 smithi176 ceph-mon[27622]: from='osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-11T21:58:42.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:42 smithi176 ceph-mon[27622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-11T21:58:42.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-mon[27727]: Adjusting osd_memory_target on smithi189 to 16064M 2024-03-11T21:58:42.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-mon[27727]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:58:42.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-mon[27727]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:58:42.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:42.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-mon[27727]: from='osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-11T21:58:42.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-mon[27727]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-11T21:58:43.377 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2[34411]: 2024-03-11T21:58:42.950+0000 7f89b8ae0700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T21:58:43.378 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:42 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2[34411]: 2024-03-11T21:58:42.954+0000 7f89b48d3700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:58:43.554 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:58:43.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:43 smithi151 ceph-mon[25773]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:43.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:43 smithi151 ceph-mon[25773]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-11T21:58:43.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:43 smithi151 ceph-mon[25773]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:58:43.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:43 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:43.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:43 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:43.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:43 smithi176 ceph-mon[27622]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:43.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:43 smithi176 ceph-mon[27622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-11T21:58:43.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:43 smithi176 ceph-mon[27622]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:58:43.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:43 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:43.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:43 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:43.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:43 smithi189 ceph-mon[27727]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:43.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:43 smithi189 ceph-mon[27727]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-11T21:58:43.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:43 smithi189 ceph-mon[27727]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:58:43.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:43 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:43.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:43 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:44 smithi151 ceph-mon[25773]: purged_snaps scrub starts 2024-03-11T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:44 smithi151 ceph-mon[25773]: purged_snaps scrub ok 2024-03-11T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:44 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/2928110761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:44 smithi151 ceph-mon[25773]: osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235] boot 2024-03-11T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:44 smithi151 ceph-mon[25773]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:58:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:44 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:44.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:44 smithi176 ceph-mon[27622]: purged_snaps scrub starts 2024-03-11T21:58:44.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:44 smithi176 ceph-mon[27622]: purged_snaps scrub ok 2024-03-11T21:58:44.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:44 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/2928110761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:58:44.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:44 smithi176 ceph-mon[27622]: osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235] boot 2024-03-11T21:58:44.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:44 smithi176 ceph-mon[27622]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:58:44.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:44 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:44.866 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1710194297,"num_in_osds":3,"osd_in_since":1710194304,"num_remapped_pgs":0} 2024-03-11T21:58:44.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:44 smithi189 ceph-mon[27727]: purged_snaps scrub starts 2024-03-11T21:58:44.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:44 smithi189 ceph-mon[27727]: purged_snaps scrub ok 2024-03-11T21:58:44.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:44 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/2928110761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:58:44.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:44 smithi189 ceph-mon[27727]: osd.2 [v2:172.21.15.189:6800/3475801235,v1:172.21.15.189:6801/3475801235] boot 2024-03-11T21:58:44.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:44 smithi189 ceph-mon[27727]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:58:44.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:44 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:58:45.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:45 smithi151 ceph-mon[25773]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:45.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:45 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:58:45.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:45 smithi176 ceph-mon[27622]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:45.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:45 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:58:45.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:58:45.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:45 smithi189 ceph-mon[27727]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:58:45.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:45 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:58:46.128 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:58:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:58:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:46 smithi151 ceph-mon[25773]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:58:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:58:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:46 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:58:46.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:46 smithi176 ceph-mon[27622]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:58:46.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:58:46.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:46 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:58:46.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:46 smithi189 ceph-mon[27727]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:58:46.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:58:46.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:46.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:46 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:47.055 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:46 smithi151 sudo[40596]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:58:47.055 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:46 smithi151 sudo[40596]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:47.055 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:46 smithi151 sudo[40596]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:47.055 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:46 smithi151 sudo[40596]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:47.369 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:47 smithi151 sudo[40599]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:58:47.369 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:47 smithi151 sudo[40599]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:47.369 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:47 smithi151 sudo[40599]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:47.369 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 21:58:47 smithi151 sudo[40599]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:47.591 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:47 smithi176 ceph-mon[27622]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:47.650 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:47 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:58:47.650 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:47 smithi176 ceph-mon[27622]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:58:47.650 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:47 smithi176 ceph-mon[27622]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:58:47.650 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:47 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:47.650 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:58:47.650 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:47.650 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34866]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:47.650 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34866]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:47 smithi151 ceph-mon[25773]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:47 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:58:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:47 smithi151 ceph-mon[25773]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:58:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:47 smithi151 ceph-mon[25773]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:58:47.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:47 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:47.847 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34869]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:58:47.847 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:47.847 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34869]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:47.847 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 21:58:47 smithi176 sudo[34869]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:47.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:47 smithi189 ceph-mon[27727]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:47.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:47 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:58:47.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:47 smithi189 ceph-mon[27727]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:58:47.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:47 smithi189 ceph-mon[27727]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:58:47.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:47 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.122 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:47 smithi189 sudo[35451]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:58:48.122 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:47 smithi189 sudo[35451]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:48.122 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:47 smithi189 sudo[35451]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:48.122 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:47 smithi189 sudo[35451]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:48.377 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35468]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:58:48.377 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35468]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:48.377 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35468]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:48.377 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35468]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:48.567 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:58:48.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40677]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:58:48.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:48.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40677]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:48.659 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40677]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:48.916 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:58:48.916 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.916 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.916 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.916 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:58:48.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 ceph-mon[27727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40695]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40695]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40695]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 sudo[40695]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:58:48.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:48.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:48 smithi151 ceph-mon[25773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:49.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:58:49.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:58:49.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:48 smithi176 ceph-mon[27622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:49.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35566]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:58:49.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:49.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35566]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:49.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:48 smithi189 sudo[35566]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:49.228 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":21,"num_osds":3,"num_up_osds":3,"osd_up_since":1710194323,"num_in_osds":3,"osd_in_since":1710194304,"num_remapped_pgs":0} 2024-03-11T21:58:49.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:58:49.463 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:58:49.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 sudo[35588]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:58:49.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 sudo[35588]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:49.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 sudo[35588]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:49.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 sudo[35588]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:49.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34954]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:58:49.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34954]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:49.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34954]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:49.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34954]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/289167445' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: Detected new or changed devices on smithi189 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: Cluster is now healthy 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:49.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:49 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:49.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:49.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/289167445' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:58:49.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:49.859 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: Detected new or changed devices on smithi189 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: Cluster is now healthy 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:49.860 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:49.861 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:49.861 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:49.861 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:49 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34978]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34978]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34978]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 sudo[34978]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/289167445' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:58:50.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: Detected new or changed devices on smithi189 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: Cluster is now healthy 2024-03-11T21:58:50.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:50.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:58:50.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:58:50.101 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:49 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:58:50.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:50 smithi151 ceph-mon[25773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:50.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:50 smithi151 ceph-mon[25773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:50.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:50 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:50 smithi151 ceph-mon[25773]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:50.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:50 smithi176 ceph-mon[27622]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:50.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:50 smithi176 ceph-mon[27622]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:50.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:50 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:50.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:50 smithi176 ceph-mon[27622]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:51.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:50 smithi189 ceph-mon[27727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:58:51.022 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:50 smithi189 ceph-mon[27727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:58:51.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:50 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:51.023 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:50 smithi189 ceph-mon[27727]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:58:51.207 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":21,"fsid":"e42c6b1c-dff1-11ee-95c8-87774f69a715","created":"2024-03-11T21:55:18.629307+0000","modified":"2024-03-11T21:58:47.478413+0000","last_up_change":"2024-03-11T21:58:43.947598+0000","last_in_change":"2024-03-11T21:58:24.003758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:58:44.727063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71f5c84c-f5c7-4613-aedc-d69504d53a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6803","nonce":265400461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6805","nonce":265400461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6809","nonce":265400461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6807","nonce":265400461}]},"public_addr":"172.21.15.151:6803/265400461","cluster_addr":"172.21.15.151:6805/265400461","heartbeat_back_addr":"172.21.15.151:6809/265400461","heartbeat_front_addr":"172.21.15.151:6807/265400461","state":["exists","up"]},{"osd":1,"uuid":"53b2450f-2382-4169-a14d-8ad8087a88f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6801","nonce":1458376516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6803","nonce":1458376516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6807","nonce":1458376516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6805","nonce":1458376516}]},"public_addr":"172.21.15.176:6801/1458376516","cluster_addr":"172.21.15.176:6803/1458376516","heartbeat_back_addr":"172.21.15.176:6807/1458376516","heartbeat_front_addr":"172.21.15.176:6805/1458376516","state":["exists","up"]},{"osd":2,"uuid":"0393c20b-b0df-49c3-b073-e73fd05e5c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6801","nonce":3475801235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6803","nonce":3475801235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6807","nonce":3475801235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6805","nonce":3475801235}]},"public_addr":"172.21.15.189:6801/3475801235","cluster_addr":"172.21.15.189:6803/3475801235","heartbeat_back_addr":"172.21.15.189:6807/3475801235","heartbeat_front_addr":"172.21.15.189:6805/3475801235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:57:48.147950+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:58:15.051999+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:58:41.967611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/369570880":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/2634633008":"2024-03-12T21:56:12.392976+0000","172.21.15.151:6801/1949257761":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/3452181919":"2024-03-12T21:55:45.632781+0000","172.21.15.151:0/299453021":"2024-03-12T21:55:45.632781+0000","172.21.15.151:0/236067974":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/4242046634":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6801/880688606":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6800/880688606":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6800/1949257761":"2024-03-12T21:56:12.392976+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:58:51.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:51 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/2499105241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:58:51.853 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:58:44.727063+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T21:58:51.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T21:58:52.003 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:51 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/2499105241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:58:52.040 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:51 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/2499105241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:58:52.091 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:58:53.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:52 smithi151 ceph-mon[25773]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:53.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:52 smithi176 ceph-mon[27622]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:53.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:52 smithi189 ceph-mon[27727]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:58:53.877 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-03-11T21:58:54.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:53 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1744011174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:58:54.240 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:53 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1744011174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:58:54.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:53 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1744011174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:58:54.498 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi151 2024-03-11T21:58:54.498 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi151=foo.a' 2024-03-11T21:58:54.752 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:58:55.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:54 smithi189 ceph-mon[27727]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-11T21:58:55.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:54 smithi151 ceph-mon[25773]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-11T21:58:55.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:54 smithi176 ceph-mon[27622]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 392 KiB/s wr, 25 op/s 2024-03-11T21:58:56.555 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled rgw.foo.a update... 2024-03-11T21:58:57.068 DEBUG:teuthology.orchestra.run.smithi151:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T21:58:57.070 INFO:tasks.cephadm:Adding prometheus.a on smithi176 2024-03-11T21:58:57.071 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi176=a' 2024-03-11T21:58:57.301 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi151=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: Saving service rgw.foo.a spec with placement smithi151=foo.a;count:1 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:58:57.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:58:57.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:57 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:57.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-11T21:58:57.686 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi151=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: Saving service rgw.foo.a spec with placement smithi151=foo.a;count:1 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.687 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:57 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 294 KiB/s wr, 18 op/s 2024-03-11T21:58:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi151=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:58:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: Saving service rgw.foo.a spec with placement smithi151=foo.a;count:1 2024-03-11T21:58:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:58:57.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:58:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:58:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:58:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:57.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:57 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:58:58.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:58 smithi176 ceph-mon[27622]: Deploying daemon rgw.foo.a on smithi151 2024-03-11T21:58:58.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:58 smithi189 ceph-mon[27727]: Deploying daemon rgw.foo.a on smithi151 2024-03-11T21:58:58.977 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:58 smithi151 ceph-mon[25773]: Deploying daemon rgw.foo.a on smithi151 2024-03-11T21:58:58.977 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:58 smithi151 systemd[1]: Starting Ceph rgw.foo.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T21:58:58.986 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled prometheus update... 2024-03-11T21:58:59.305 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:58 smithi151 podman[41439]: 2024-03-11 21:58:58.983520565 +0000 UTC m=+0.108797102 container create b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:58:59.306 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:58 smithi151 podman[41439]: 2024-03-11 21:58:58.89264355 +0000 UTC m=+0.017920098 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:58:59.306 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:59 smithi151 podman[41439]: 2024-03-11 21:58:59.227608988 +0000 UTC m=+0.352885526 container init b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T21:58:59.306 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:59 smithi151 podman[41439]: 2024-03-11 21:58:59.232677846 +0000 UTC m=+0.357954383 container start b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:58:59.538 DEBUG:teuthology.orchestra.run.smithi176:prometheus.a> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T21:58:59.542 INFO:tasks.cephadm:Adding node-exporter.a on smithi151 2024-03-11T21:58:59.542 INFO:tasks.cephadm:Adding node-exporter.b on smithi176 2024-03-11T21:58:59.542 INFO:tasks.cephadm:Adding node-exporter.c on smithi189 2024-03-11T21:58:59.542 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi151=a;smithi176=b;smithi189=c' 2024-03-11T21:58:59.779 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:58:59.805 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:59 smithi151 bash[41439]: b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb 2024-03-11T21:58:59.805 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 21:58:59 smithi151 systemd[1]: Started Ceph rgw.foo.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:58:59.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 235 KiB/s wr, 15 op/s 2024-03-11T21:58:59.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:58:59 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 235 KiB/s wr, 15 op/s 2024-03-11T21:58:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:58:59 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 235 KiB/s wr, 15 op/s 2024-03-11T21:58:59.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:58:59.878 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:58:59 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:00.660 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:00.660 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: Saving service prometheus spec with placement smithi176=a;count:1 2024-03-11T21:59:00.660 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: Saving service rgw.foo.a spec with placement smithi151=foo.a;count:1 2024-03-11T21:59:00.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:00.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:59:00.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:59:00.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:59:00.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/3193079137' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:59:00.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:00 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/59203575' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: Saving service prometheus spec with placement smithi176=a;count:1 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: Saving service rgw.foo.a spec with placement smithi151=foo.a;count:1 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:59:00.835 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:59:00.836 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/3193079137' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:59:00.836 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:00 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/59203575' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: Saving service prometheus spec with placement smithi176=a;count:1 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: Saving service rgw.foo.a spec with placement smithi151=foo.a;count:1 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:59:00.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/3193079137' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:59:00.850 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:00 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/59203575' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:59:01.641 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled node-exporter update... 2024-03-11T21:59:01.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:01 smithi151 ceph-mon[25773]: Checking dashboard <-> RGW credentials 2024-03-11T21:59:01.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:01 smithi151 ceph-mon[25773]: pgmap v99: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 235 KiB/s wr, 15 op/s 2024-03-11T21:59:01.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:01 smithi151 ceph-mon[25773]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:59:01.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:01 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/3193079137' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:59:01.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:01 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/59203575' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:59:01.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:01 smithi151 ceph-mon[25773]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:59:01.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:01 smithi189 ceph-mon[27727]: Checking dashboard <-> RGW credentials 2024-03-11T21:59:01.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:01 smithi189 ceph-mon[27727]: pgmap v99: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 235 KiB/s wr, 15 op/s 2024-03-11T21:59:01.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:01 smithi189 ceph-mon[27727]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:59:01.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:01 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/3193079137' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:59:01.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:01 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/59203575' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:59:01.925 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:01 smithi189 ceph-mon[27727]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:59:02.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:01 smithi176 ceph-mon[27622]: Checking dashboard <-> RGW credentials 2024-03-11T21:59:02.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:01 smithi176 ceph-mon[27622]: pgmap v99: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 235 KiB/s wr, 15 op/s 2024-03-11T21:59:02.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:01 smithi176 ceph-mon[27622]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:59:02.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:01 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/3193079137' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:59:02.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:01 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/59203575' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:59:02.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:01 smithi176 ceph-mon[27622]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:59:02.201 DEBUG:teuthology.orchestra.run.smithi151:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T21:59:02.204 DEBUG:teuthology.orchestra.run.smithi176:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T21:59:02.207 DEBUG:teuthology.orchestra.run.smithi189:node-exporter.c> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-11T21:59:02.209 INFO:tasks.cephadm:Adding alertmanager.a on smithi151 2024-03-11T21:59:02.209 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi151=a' 2024-03-11T21:59:02.480 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:59:02.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi151=a;smithi176=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:02.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: Saving service node-exporter spec with placement smithi151=a;smithi176=b;smithi189=c;count:3 2024-03-11T21:59:02.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:02.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:59:02.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:59:02.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:59:02.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: pgmap v102: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:02.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:02 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi151=a;smithi176=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: Saving service node-exporter spec with placement smithi151=a;smithi176=b;smithi189=c;count:3 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: pgmap v102: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:02.852 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:02 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi151=a;smithi176=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: Saving service node-exporter spec with placement smithi151=a;smithi176=b;smithi189=c;count:3 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:59:03.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: pgmap v102: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:03.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:02 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.018 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled alertmanager update... 2024-03-11T21:59:04.018 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.018 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:59:04.018 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:59:04.018 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:59:04.019 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.019 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.019 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:03 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:03 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:59:04.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:59:04.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:59:04.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:03 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:04.556 DEBUG:teuthology.orchestra.run.smithi151:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T21:59:04.559 INFO:tasks.cephadm:Adding grafana.a on smithi176 2024-03-11T21:59:04.559 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi176=a' 2024-03-11T21:59:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:59:05.163 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:05.163 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: Saving service alertmanager spec with placement smithi151=a;count:1 2024-03-11T21:59:05.163 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:05.163 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:59:05.163 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:59:05.164 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:59:05.164 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:05 smithi189 ceph-mon[27727]: pgmap v105: 97 pgs: 11 active+clean, 19 creating+peering, 67 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: Saving service alertmanager spec with placement smithi151=a;count:1 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:59:05.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:05 smithi151 ceph-mon[25773]: pgmap v105: 97 pgs: 11 active+clean, 19 creating+peering, 67 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: Saving service alertmanager spec with placement smithi151=a;count:1 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:59:05.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:05 smithi176 ceph-mon[27622]: pgmap v105: 97 pgs: 11 active+clean, 19 creating+peering, 67 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:06.161 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:06 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:59:06.161 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:06 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:59:06.162 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:06 smithi189 ceph-mon[27727]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:59:06.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:06 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:59:06.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:06 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:59:06.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:06 smithi151 ceph-mon[25773]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:59:06.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:06 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:59:06.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:06 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:59:06.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:06 smithi176 ceph-mon[27622]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:59:06.617 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled grafana update... 2024-03-11T21:59:07.123 DEBUG:teuthology.orchestra.run.smithi176:grafana.a> sudo journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T21:59:07.125 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T21:59:07.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:59:07.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:07 smithi151 ceph-mon[25773]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:59:07.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:07 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:59:07.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:07 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:59:07.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:07 smithi151 ceph-mon[25773]: pgmap v108: 129 pgs: 65 active+clean, 23 creating+peering, 41 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T21:59:07.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:07 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:07.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:07 smithi176 ceph-mon[27622]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:59:07.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:07 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:59:07.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:07 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:59:07.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:07 smithi176 ceph-mon[27622]: pgmap v108: 129 pgs: 65 active+clean, 23 creating+peering, 41 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T21:59:07.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:07 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:07.373 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:07.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:07 smithi189 ceph-mon[27727]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:59:07.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:07 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:59:07.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:07 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:59:07.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:07 smithi189 ceph-mon[27727]: pgmap v108: 129 pgs: 65 active+clean, 23 creating+peering, 41 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T21:59:07.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:07 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:08.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: Saving service grafana spec with placement smithi176=a;count:1 2024-03-11T21:59:08.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:59:08.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:59:08.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:59:08.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:59:08.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:59:08.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:08 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: Saving service grafana spec with placement smithi176=a;count:1 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:59:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:08 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:59:08.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: Saving service grafana spec with placement smithi176=a;count:1 2024-03-11T21:59:08.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:59:08.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:59:08.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:59:08.372 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:59:08.373 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:59:08.373 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.373 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:08.373 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:08 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: pgmap v111: 129 pgs: 98 active+clean, 12 creating+peering, 19 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: Deploying daemon prometheus.a on smithi176 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:09 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: pgmap v111: 129 pgs: 98 active+clean, 12 creating+peering, 19 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-11T21:59:09.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: Deploying daemon prometheus.a on smithi176 2024-03-11T21:59:09.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.350 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:09 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/4203438446' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1961389437' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: pgmap v111: 129 pgs: 98 active+clean, 12 creating+peering, 19 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: Deploying daemon prometheus.a on smithi176 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:09 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:09.455 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-03-11T21:59:09.455 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCtfu9lFnloGxAA/exwFCy0CTzldED5lvxt6g== 2024-03-11T21:59:10.010 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T21:59:10.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T21:59:10.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T21:59:10.052 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:59:10.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:10 smithi151 ceph-mon[25773]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:59:10.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:10 smithi151 ceph-mon[25773]: Cluster is now healthy 2024-03-11T21:59:10.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:10 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/353634526' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:10.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:10 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/353634526' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:10.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:10 smithi176 ceph-mon[27622]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:59:10.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:10 smithi176 ceph-mon[27622]: Cluster is now healthy 2024-03-11T21:59:10.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:10 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/353634526' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:10.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:10 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/353634526' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:10.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:10 smithi189 ceph-mon[27727]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:59:10.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:10 smithi189 ceph-mon[27727]: Cluster is now healthy 2024-03-11T21:59:10.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:10 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/353634526' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:10.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:10 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/353634526' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:10.621 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:59:11.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:11 smithi176 ceph-mon[27622]: pgmap v112: 129 pgs: 119 active+clean, 10 unknown; 579 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 26 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-03-11T21:59:11.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:11 smithi189 ceph-mon[27727]: pgmap v112: 129 pgs: 119 active+clean, 10 unknown; 579 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 26 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-03-11T21:59:11.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:11 smithi151 ceph-mon[25773]: pgmap v112: 129 pgs: 119 active+clean, 10 unknown; 579 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 26 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-03-11T21:59:13.747 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-03-11T21:59:13.747 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCxfu9l/2iHLBAAolU9380mKQbWMPfP95rK9g== 2024-03-11T21:59:13.748 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:13 smithi176 ceph-mon[27622]: pgmap v113: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 48 KiB/s rd, 3.9 KiB/s wr, 114 op/s 2024-03-11T21:59:13.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:13 smithi151 ceph-mon[25773]: pgmap v113: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 48 KiB/s rd, 3.9 KiB/s wr, 114 op/s 2024-03-11T21:59:13.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:13 smithi189 ceph-mon[27727]: pgmap v113: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 48 KiB/s rd, 3.9 KiB/s wr, 114 op/s 2024-03-11T21:59:14.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:14 smithi176 ceph-mon[27622]: from='client.? 172.21.15.176:0/2662806602' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:14.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:14 smithi176 ceph-mon[27622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:14.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:14 smithi176 ceph-mon[27622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:14.633 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T21:59:14.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T21:59:14.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T21:59:14.677 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:59:14.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:14 smithi151 ceph-mon[25773]: from='client.? 172.21.15.176:0/2662806602' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:14.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:14 smithi151 ceph-mon[25773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:14.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:14 smithi151 ceph-mon[25773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:14 smithi189 ceph-mon[27727]: from='client.? 172.21.15.176:0/2662806602' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:14 smithi189 ceph-mon[27727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:14.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:14 smithi189 ceph-mon[27727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:14.930 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:59:15.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:15 smithi151 ceph-mon[25773]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-11T21:59:15.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:15 smithi176 ceph-mon[27622]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-11T21:59:15.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:15 smithi189 ceph-mon[27727]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-11T21:59:16.941 INFO:teuthology.orchestra.run.smithi189.stdout:[client.2] 2024-03-11T21:59:16.941 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQC0fu9lSTkgOBAAaNWXPdoQ4KSxQJcZ55ZLaA== 2024-03-11T21:59:17.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:17 smithi189 ceph-mon[27727]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 49 KiB/s rd, 3.7 KiB/s wr, 120 op/s 2024-03-11T21:59:17.488 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:17 smithi189 ceph-mon[27727]: from='client.? 172.21.15.189:0/1186008940' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:17.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:17 smithi189 ceph-mon[27727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:17.489 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:17 smithi189 ceph-mon[27727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:17.514 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T21:59:17.515 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-11T21:59:17.515 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-11T21:59:17.557 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T21:59:17.557 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T21:59:17.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T21:59:17.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:17 smithi151 ceph-mon[25773]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 49 KiB/s rd, 3.7 KiB/s wr, 120 op/s 2024-03-11T21:59:17.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:17 smithi151 ceph-mon[25773]: from='client.? 172.21.15.189:0/1186008940' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:17.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:17 smithi151 ceph-mon[25773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:17.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:17 smithi151 ceph-mon[25773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:17.806 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:17.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:17 smithi176 ceph-mon[27622]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 49 KiB/s rd, 3.7 KiB/s wr, 120 op/s 2024-03-11T21:59:17.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:17 smithi176 ceph-mon[27622]: from='client.? 172.21.15.189:0/1186008940' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:17.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:17 smithi176 ceph-mon[27622]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:59:17.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:17 smithi176 ceph-mon[27622]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:59:19.698 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:59:19.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:19 smithi151 ceph-mon[25773]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 107 op/s 2024-03-11T21:59:19.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:19 smithi176 ceph-mon[27622]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 107 op/s 2024-03-11T21:59:19.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:19 smithi189 ceph-mon[27727]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 107 op/s 2024-03-11T21:59:20.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:20 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1353980896' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:59:20.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:20 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1353980896' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:59:20.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:20 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1353980896' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:59:20.997 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":3807166017},{"type":"v1","addr":"172.21.15.151:6801","nonce":3807166017}]},"active_addr":"172.21.15.151:6801/3807166017","active_change":"2024-03-11T21:56:12.393119+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.151:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":1836832413}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3191014057}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":424838248}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":4109294992}]}]} 2024-03-11T21:59:21.003 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T21:59:21.003 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:59:21.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:59:21.391 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:21.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:21 smithi151 ceph-mon[25773]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-03-11T21:59:21.778 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:21 smithi189 ceph-mon[27727]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-03-11T21:59:21.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:21 smithi176 ceph-mon[27622]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-03-11T21:59:23.706 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:23 smithi151 ceph-mon[25773]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 27 KiB/s rd, 1.8 KiB/s wr, 66 op/s 2024-03-11T21:59:23.725 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:23 smithi176 ceph-mon[27622]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 27 KiB/s rd, 1.8 KiB/s wr, 66 op/s 2024-03-11T21:59:23.754 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:59:23.755 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":30,"fsid":"e42c6b1c-dff1-11ee-95c8-87774f69a715","created":"2024-03-11T21:55:18.629307+0000","modified":"2024-03-11T21:59:08.057793+0000","last_up_change":"2024-03-11T21:58:43.947598+0000","last_in_change":"2024-03-11T21:58:24.003758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:58:44.727063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:58:59.507469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:59:01.340811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:59:03.289254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:59:05.345533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71f5c84c-f5c7-4613-aedc-d69504d53a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6803","nonce":265400461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6805","nonce":265400461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6809","nonce":265400461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6807","nonce":265400461}]},"public_addr":"172.21.15.151:6803/265400461","cluster_addr":"172.21.15.151:6805/265400461","heartbeat_back_addr":"172.21.15.151:6809/265400461","heartbeat_front_addr":"172.21.15.151:6807/265400461","state":["exists","up"]},{"osd":1,"uuid":"53b2450f-2382-4169-a14d-8ad8087a88f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6801","nonce":1458376516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6803","nonce":1458376516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6807","nonce":1458376516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6805","nonce":1458376516}]},"public_addr":"172.21.15.176:6801/1458376516","cluster_addr":"172.21.15.176:6803/1458376516","heartbeat_back_addr":"172.21.15.176:6807/1458376516","heartbeat_front_addr":"172.21.15.176:6805/1458376516","state":["exists","up"]},{"osd":2,"uuid":"0393c20b-b0df-49c3-b073-e73fd05e5c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6801","nonce":3475801235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6803","nonce":3475801235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6807","nonce":3475801235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6805","nonce":3475801235}]},"public_addr":"172.21.15.189:6801/3475801235","cluster_addr":"172.21.15.189:6803/3475801235","heartbeat_back_addr":"172.21.15.189:6807/3475801235","heartbeat_front_addr":"172.21.15.189:6805/3475801235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:57:48.147950+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:58:15.051999+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:58:41.967611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/369570880":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/2634633008":"2024-03-12T21:56:12.392976+0000","172.21.15.151:6801/1949257761":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/3452181919":"2024-03-12T21:55:45.632781+0000","172.21.15.151:0/299453021":"2024-03-12T21:55:45.632781+0000","172.21.15.151:0/236067974":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/4242046634":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6801/880688606":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6800/880688606":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6800/1949257761":"2024-03-12T21:56:12.392976+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:59:23.877 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:23 smithi189 ceph-mon[27727]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 27 KiB/s rd, 1.8 KiB/s wr, 66 op/s 2024-03-11T21:59:24.348 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:59:24.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:59:24.591 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/918014202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:24.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:24 smithi151 ceph-mon[25773]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 13 KiB/s rd, 853 B/s wr, 32 op/s 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/918014202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.091 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.092 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:24 smithi176 ceph-mon[27622]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 13 KiB/s rd, 853 B/s wr, 32 op/s 2024-03-11T21:59:25.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/918014202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:59:25.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:25.128 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:24 smithi189 ceph-mon[27727]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 13 KiB/s rd, 853 B/s wr, 32 op/s 2024-03-11T21:59:25.993 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:25 smithi176 podman[36077]: 2024-03-11 21:59:25.900098441 +0000 UTC m=+0.125807793 container create b079a4546be4adb16b5b0e100beb628db64800abb03240a4128cafc210e5800f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:59:26.221 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:59:26.221 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":30,"fsid":"e42c6b1c-dff1-11ee-95c8-87774f69a715","created":"2024-03-11T21:55:18.629307+0000","modified":"2024-03-11T21:59:08.057793+0000","last_up_change":"2024-03-11T21:58:43.947598+0000","last_in_change":"2024-03-11T21:58:24.003758+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:58:44.727063+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:58:59.507469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:59:01.340811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:59:03.289254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:59:05.345533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"71f5c84c-f5c7-4613-aedc-d69504d53a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6803","nonce":265400461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6805","nonce":265400461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6809","nonce":265400461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":265400461},{"type":"v1","addr":"172.21.15.151:6807","nonce":265400461}]},"public_addr":"172.21.15.151:6803/265400461","cluster_addr":"172.21.15.151:6805/265400461","heartbeat_back_addr":"172.21.15.151:6809/265400461","heartbeat_front_addr":"172.21.15.151:6807/265400461","state":["exists","up"]},{"osd":1,"uuid":"53b2450f-2382-4169-a14d-8ad8087a88f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6801","nonce":1458376516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6803","nonce":1458376516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6807","nonce":1458376516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":1458376516},{"type":"v1","addr":"172.21.15.176:6805","nonce":1458376516}]},"public_addr":"172.21.15.176:6801/1458376516","cluster_addr":"172.21.15.176:6803/1458376516","heartbeat_back_addr":"172.21.15.176:6807/1458376516","heartbeat_front_addr":"172.21.15.176:6805/1458376516","state":["exists","up"]},{"osd":2,"uuid":"0393c20b-b0df-49c3-b073-e73fd05e5c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6801","nonce":3475801235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6803","nonce":3475801235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6807","nonce":3475801235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3475801235},{"type":"v1","addr":"172.21.15.189:6805","nonce":3475801235}]},"public_addr":"172.21.15.189:6801/3475801235","cluster_addr":"172.21.15.189:6803/3475801235","heartbeat_back_addr":"172.21.15.189:6807/3475801235","heartbeat_front_addr":"172.21.15.189:6805/3475801235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:57:48.147950+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:58:15.051999+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:58:41.967611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/369570880":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/2634633008":"2024-03-12T21:56:12.392976+0000","172.21.15.151:6801/1949257761":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/3452181919":"2024-03-12T21:55:45.632781+0000","172.21.15.151:0/299453021":"2024-03-12T21:55:45.632781+0000","172.21.15.151:0/236067974":"2024-03-12T21:56:12.392976+0000","172.21.15.151:0/4242046634":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6801/880688606":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6800/880688606":"2024-03-12T21:55:45.632781+0000","172.21.15.151:6800/1949257761":"2024-03-12T21:56:12.392976+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:59:26.470 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/2030759246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:59:26.470 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 podman[36077]: 2024-03-11 21:59:26.119185717 +0000 UTC m=+0.344895095 container init b079a4546be4adb16b5b0e100beb628db64800abb03240a4128cafc210e5800f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:59:26.470 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 podman[36077]: 2024-03-11 21:59:26.126586442 +0000 UTC m=+0.352295798 container start b079a4546be4adb16b5b0e100beb628db64800abb03240a4128cafc210e5800f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.151Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.151Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.151Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi176 (none))" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.151Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.151Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.152Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.152Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.153Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.153Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.156Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.156Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.435µs 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.156Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:59:26.471 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.157Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.157Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.659µs wal_replay_duration=392.178µs wbl_replay_duration=123ns total_replay_duration=450.09µs 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.158Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.158Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.158Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.179Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.403466ms db_storage=917ns remote_storage=1.764µs web_handler=476ns query_engine=620ns scrape=619.497µs scrape_sd=128.606µs notify=1.051µs notify_sd=1.943µs rules=20.396947ms tracing=6.983µs 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.179Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T21:59:26.179Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 bash[36077]: b079a4546be4adb16b5b0e100beb628db64800abb03240a4128cafc210e5800f 2024-03-11T21:59:26.472 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 21:59:26 smithi176 systemd[1]: Started Ceph prometheus.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:59:26.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:26 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/2030759246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:59:26.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:26 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/2030759246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:59:26.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:59:26.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:59:26.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:59:27.046 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:27.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:27.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:27.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:59:27.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-mon[25773]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:27.329 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.329 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.329 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.329 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:59:27.329 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-mon[27622]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:27.581 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ignoring --setuser ceph since I am not root 2024-03-11T21:59:27.582 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ignoring --setgroup ceph since I am not root 2024-03-11T21:59:27.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: ignoring --setuser ceph since I am not root 2024-03-11T21:59:27.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: ignoring --setgroup ceph since I am not root 2024-03-11T21:59:27.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-mgr[29339]: -- 172.21.15.176:0/2349636241 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dbe74e41a0 con 0x55dbe74bac00 2024-03-11T21:59:27.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' 2024-03-11T21:59:27.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:59:27.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-mon[27727]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:27.628 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: ignoring --setuser ceph since I am not root 2024-03-11T21:59:27.628 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: ignoring --setgroup ceph since I am not root 2024-03-11T21:59:27.628 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-mgr[30216]: -- 172.21.15.189:0/3295186293 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5623d487a1a0 con 0x5623d4851000 2024-03-11T21:59:27.972 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:27.883+0000 7f92aa443200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:59:27.972 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:27 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:27.977+0000 7f92aa443200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:59:28.129 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:27.861+0000 7f0813684200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:59:28.129 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:27 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:27.954+0000 7f0813684200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:59:28.293 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:27.900+0000 7fa418632200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:59:28.293 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:27 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:27.995+0000 7fa418632200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:59:28.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:28 smithi151 ceph-mon[25773]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:59:28.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:28 smithi151 ceph-mon[25773]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:59:28.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:28 smithi176 ceph-mon[27622]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:59:28.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:28 smithi176 ceph-mon[27622]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:59:28.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:28 smithi189 ceph-mon[27727]: from='mgr.14152 172.21.15.151:0/2685586774' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:59:28.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:28 smithi189 ceph-mon[27727]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:59:28.753 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:28 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:28.661+0000 7f92aa443200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:59:28.877 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:28 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:28.608+0000 7f0813684200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:59:28.877 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:28 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:28.711+0000 7f0813684200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:59:28.877 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:28 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:28.853+0000 7f0813684200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:59:28.896 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:28 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:28.649+0000 7fa418632200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:59:28.896 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:28 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:28.743+0000 7fa418632200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:59:28.896 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:28 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:28.901+0000 7fa418632200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:59:29.055 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:28 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:28.759+0000 7f92aa443200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:59:29.055 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:28 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:28.905+0000 7f92aa443200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:59:29.055 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.005+0000 7f92aa443200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:59:29.166 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.004+0000 7fa418632200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:59:29.166 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.088+0000 7fa418632200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:59:29.294 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:28 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:28.950+0000 7f0813684200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:59:29.294 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:29.032+0000 7f0813684200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:59:29.294 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:29.300+0000 7f0813684200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:59:29.374 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.096+0000 7f92aa443200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:59:29.469 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.364+0000 7fa418632200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:59:29.469 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.475+0000 7fa418632200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:59:29.583 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:29.409+0000 7f0813684200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:59:29.583 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:29.494+0000 7f0813684200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:59:29.583 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:29.589+0000 7f0813684200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:59:29.671 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.379+0000 7f92aa443200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:59:29.671 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.492+0000 7f92aa443200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:59:29.671 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.580+0000 7f92aa443200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:59:29.835 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.562+0000 7fa418632200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:59:29.835 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.661+0000 7fa418632200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:59:29.835 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:29 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:29.840+0000 7fa418632200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:59:29.877 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:29 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:29.765+0000 7f0813684200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:59:30.008 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.677+0000 7f92aa443200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:59:30.008 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:29 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:29.857+0000 7f92aa443200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:59:30.942 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:30.857+0000 7f92aa443200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:59:31.035 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:30 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:30.743+0000 7f0813684200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:59:31.035 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:30 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:30.876+0000 7f0813684200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:59:31.099 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:30 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:30.796+0000 7fa418632200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:59:31.099 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:30 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:30.927+0000 7fa418632200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:59:31.305 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:31 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:31.000+0000 7f92aa443200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:59:31.306 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:31 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:31.220+0000 7f0813684200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:59:31.306 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:31 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:31.311+0000 7f0813684200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:59:31.543 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:31.264+0000 7fa418632200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:59:31.544 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:31.354+0000 7fa418632200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:59:31.544 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:31.549+0000 7fa418632200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:59:31.621 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:31 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:31.354+0000 7f92aa443200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:59:31.621 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:31 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:31.447+0000 7f92aa443200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:59:31.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:31 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:31.512+0000 7f0813684200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:59:31.649 INFO:teuthology.orchestra.run.smithi151.stdout:55834574865 2024-03-11T21:59:31.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:59:31.849 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:31 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:31.737+0000 7fa418632200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:59:31.903 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:31 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:31.647+0000 7f92aa443200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:59:31.904 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:31 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:31.847+0000 7f92aa443200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:59:31.999 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:31 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:31.704+0000 7f0813684200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:59:32.161 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:32 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:32.147+0000 7f92aa443200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:59:32.272 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:32 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:32.004+0000 7f0813684200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:59:32.272 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:32 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:32.278+0000 7f0813684200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:59:32.276 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:32.020+0000 7fa418632200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:59:32.276 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:32.281+0000 7fa418632200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:59:32.410 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:32 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:32.413+0000 7f92aa443200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:59:32.437 INFO:teuthology.orchestra.run.smithi151.stdout:77309411339 2024-03-11T21:59:32.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:59:32.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:32.517+0000 7fa418632200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:59:32.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:32 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:32.521+0000 7f0813684200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:59:32.906 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:33.005 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:32 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:32.669+0000 7f92aa443200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:59:33.325 INFO:teuthology.orchestra.run.smithi151.stdout:34359738391 2024-03-11T21:59:33.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:59:33.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:33 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:33.534+0000 7f92aa443200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:59:33.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:33 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:33.319+0000 7fa418632200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:59:33.612 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:33.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:33 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:33.338+0000 7f0813684200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:59:34.177 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:34.136+0000 7f92aa443200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:59:34.201 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:33 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:33.893+0000 7fa418632200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:59:34.201 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:33 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:33.991+0000 7fa418632200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:59:34.201 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:34.207+0000 7fa418632200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:59:34.206 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:33 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:33.903+0000 7f0813684200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:59:34.207 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:33.999+0000 7f0813684200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:59:34.207 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:34.212+0000 7f0813684200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:34.238+0000 7f92aa443200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: Standby manager daemon x restarted 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: Standby manager daemon x started 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: Standby manager daemon z restarted 2024-03-11T21:59:34.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: Standby manager daemon z started 2024-03-11T21:59:34.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:59:34.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:59:34.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:59:34.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:59:34.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-mon[25773]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: [11/Mar/2024:21:59:34] ENGINE Bus STARTING 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: CherryPy Checker: 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: The Application mounted at '' has an empty config. 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: [11/Mar/2024:21:59:34] ENGINE Serving on http://:::9283 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x[29316]: [11/Mar/2024:21:59:34] ENGINE Bus STARTED 2024-03-11T21:59:34.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: Standby manager daemon x restarted 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: Standby manager daemon x started 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: Standby manager daemon z restarted 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: Standby manager daemon z started 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:59:34.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:34 smithi176 ceph-mon[27622]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:59:34.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: [11/Mar/2024:21:59:34] ENGINE Bus STARTING 2024-03-11T21:59:34.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: CherryPy Checker: 2024-03-11T21:59:34.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: The Application mounted at '' has an empty config. 2024-03-11T21:59:34.627 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: [11/Mar/2024:21:59:34] ENGINE Serving on http://:::9283 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z[30141]: [11/Mar/2024:21:59:34] ENGINE Bus STARTED 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: Standby manager daemon x restarted 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: Standby manager daemon x started 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: Standby manager daemon z restarted 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: Standby manager daemon z started 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.176:0/3554334071' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:59:34.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:59:34.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:59:34.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:34 smithi189 ceph-mon[27727]: from='mgr.? 172.21.15.189:0/1615733273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:59:34.713 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:34.467+0000 7f92aa443200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:59:34.713 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:34] ENGINE Bus STARTING 2024-03-11T21:59:34.713 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: CherryPy Checker: 2024-03-11T21:59:34.713 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: The Application mounted at '' has an empty config. 2024-03-11T21:59:34.713 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T21:59:34.714 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:34] ENGINE Serving on http://:::9283 2024-03-11T21:59:34.714 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:34] ENGINE Bus STARTED 2024-03-11T21:59:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:35.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: Active manager daemon y restarted 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: Activating manager daemon y 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: mgrmap e19: y(active, starting, since 0.0101055s), standbys: x, z 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:59:35.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: Manager daemon y is now available 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:59:35.558 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:59:35.558 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:59:35.558 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:59:35.558 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:35 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: Active manager daemon y restarted 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: Activating manager daemon y 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: mgrmap e19: y(active, starting, since 0.0101055s), standbys: x, z 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:59:35.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: Manager daemon y is now available 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:35.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:59:35.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:35 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:59:35.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T21:59:35.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: Active manager daemon y restarted 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: Activating manager daemon y 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: mgrmap e19: y(active, starting, since 0.0101055s), standbys: x, z 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:59:35.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: Manager daemon y is now available 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]': finished 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:59:35.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:59:35.630 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:35 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:59:36.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:59:36.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:59:36.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:59:36.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: [11/Mar/2024:21:59:34] ENGINE Bus STARTING 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: [11/Mar/2024:21:59:35] ENGINE Serving on https://172.21.15.151:7151 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: [11/Mar/2024:21:59:35] ENGINE Serving on http://172.21.15.151:8765 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: [11/Mar/2024:21:59:35] ENGINE Bus STARTED 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: mgrmap e20: y(active, since 1.01551s), standbys: x, z 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:36 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: [11/Mar/2024:21:59:34] ENGINE Bus STARTING 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: [11/Mar/2024:21:59:35] ENGINE Serving on https://172.21.15.151:7151 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: [11/Mar/2024:21:59:35] ENGINE Serving on http://172.21.15.151:8765 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: [11/Mar/2024:21:59:35] ENGINE Bus STARTED 2024-03-11T21:59:36.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: mgrmap e20: y(active, since 1.01551s), standbys: x, z 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:36 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-11T21:59:36.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-11T21:59:36.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-11T21:59:36.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: [11/Mar/2024:21:59:34] ENGINE Bus STARTING 2024-03-11T21:59:36.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: [11/Mar/2024:21:59:35] ENGINE Serving on https://172.21.15.151:7151 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: [11/Mar/2024:21:59:35] ENGINE Serving on http://172.21.15.151:8765 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: [11/Mar/2024:21:59:35] ENGINE Bus STARTED 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: mgrmap e20: y(active, since 1.01551s), standbys: x, z 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:36.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:36 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:37.251 INFO:teuthology.orchestra.run.smithi151.stdout:77309411340 2024-03-11T21:59:37.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:37.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: Deploying daemon alertmanager.a on smithi151 2024-03-11T21:59:37.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:37 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/431641027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:59:37.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:37.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: Deploying daemon alertmanager.a on smithi151 2024-03-11T21:59:37.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:37 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/431641027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:59:37.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Updating smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Updating smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Updating smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: Deploying daemon alertmanager.a on smithi151 2024-03-11T21:59:37.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:37 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/431641027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:59:37.870 INFO:teuthology.orchestra.run.smithi151.stdout:34359738392 2024-03-11T21:59:38.067 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411340 for osd.2 2024-03-11T21:59:38.067 DEBUG:teuthology.parallel:result is None 2024-03-11T21:59:38.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:38 smithi151 ceph-mon[25773]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:38.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:38 smithi151 ceph-mon[25773]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:59:38.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:38 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1605265802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:59:38.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:38 smithi176 ceph-mon[27622]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:38.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:38 smithi176 ceph-mon[27622]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:59:38.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:38 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1605265802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:59:38.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:38 smithi189 ceph-mon[27727]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:38.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:38 smithi189 ceph-mon[27727]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:59:38.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:38 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1605265802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:59:38.786 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738391 got 34359738392 for osd.0 2024-03-11T21:59:38.871 DEBUG:teuthology.parallel:result is None 2024-03-11T21:59:39.379 INFO:teuthology.orchestra.run.smithi151.stdout:55834574866 2024-03-11T21:59:40.147 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574866 for osd.1 2024-03-11T21:59:40.147 DEBUG:teuthology.parallel:result is None 2024-03-11T21:59:40.147 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:59:40.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:59:40.396 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:40.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:40 smithi151 ceph-mon[25773]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:40.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:40 smithi151 ceph-mon[25773]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:59:40.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:40 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1481068361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:59:40.518 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:40 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ::ffff:172.21.15.176 - - [11/Mar/2024:21:59:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:59:40.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:40 smithi176 ceph-mon[27622]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:40.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:40 smithi176 ceph-mon[27622]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:59:40.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:40 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1481068361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:59:40.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:40 smithi189 ceph-mon[27727]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:40.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:40 smithi189 ceph-mon[27727]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:59:40.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:40 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1481068361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:59:42.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:42 smithi151 ceph-mon[25773]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:42.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:42 smithi176 ceph-mon[27622]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:42.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:42 smithi189 ceph-mon[27727]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:59:43.155 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-03-11T21:59:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:44 smithi151 ceph-mon[25773]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:59:44.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:44 smithi151 ceph-mon[25773]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:59:44.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:44 smithi176 ceph-mon[27622]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:59:44.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:44 smithi176 ceph-mon[27622]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:59:44.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:44 smithi189 ceph-mon[27727]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:59:44.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:44 smithi189 ceph-mon[27727]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:59:44.876 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-03-11T21:59:42.497192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.999030"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487575+0000","last_change":"2024-03-11T21:59:07.075476+0000","last_active":"2024-03-11T21:59:34.487575+0000","last_peered":"2024-03-11T21:59:34.487575+0000","last_clean":"2024-03-11T21:59:34.487575+0000","last_became_active":"2024-03-11T21:59:07.075187+0000","last_became_peered":"2024-03-11T21:59:07.075187+0000","last_unstale":"2024-03-11T21:59:34.487575+0000","last_undegraded":"2024-03-11T21:59:34.487575+0000","last_fullsized":"2024-03-11T21:59:34.487575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:45:23.775068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487906+0000","last_change":"2024-03-11T21:59:05.051540+0000","last_active":"2024-03-11T21:59:34.487906+0000","last_peered":"2024-03-11T21:59:34.487906+0000","last_clean":"2024-03-11T21:59:34.487906+0000","last_became_active":"2024-03-11T21:59:05.051224+0000","last_became_peered":"2024-03-11T21:59:05.051224+0000","last_unstale":"2024-03-11T21:59:34.487906+0000","last_undegraded":"2024-03-11T21:59:34.487906+0000","last_fullsized":"2024-03-11T21:59:34.487906+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:09:46.632361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489396+0000","last_change":"2024-03-11T21:59:01.040354+0000","last_active":"2024-03-11T21:59:34.489396+0000","last_peered":"2024-03-11T21:59:34.489396+0000","last_clean":"2024-03-11T21:59:34.489396+0000","last_became_active":"2024-03-11T21:59:01.040045+0000","last_became_peered":"2024-03-11T21:59:01.040045+0000","last_unstale":"2024-03-11T21:59:34.489396+0000","last_undegraded":"2024-03-11T21:59:34.489396+0000","last_fullsized":"2024-03-11T21:59:34.489396+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:36:13.634549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488109+0000","last_change":"2024-03-11T21:59:03.041716+0000","last_active":"2024-03-11T21:59:34.488109+0000","last_peered":"2024-03-11T21:59:34.488109+0000","last_clean":"2024-03-11T21:59:34.488109+0000","last_became_active":"2024-03-11T21:59:03.041389+0000","last_became_peered":"2024-03-11T21:59:03.041389+0000","last_unstale":"2024-03-11T21:59:34.488109+0000","last_undegraded":"2024-03-11T21:59:34.488109+0000","last_fullsized":"2024-03-11T21:59:34.488109+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:44:53.468648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486479+0000","last_change":"2024-03-11T21:59:07.079058+0000","last_active":"2024-03-11T21:59:34.486479+0000","last_peered":"2024-03-11T21:59:34.486479+0000","last_clean":"2024-03-11T21:59:34.486479+0000","last_became_active":"2024-03-11T21:59:07.078425+0000","last_became_peered":"2024-03-11T21:59:07.078425+0000","last_unstale":"2024-03-11T21:59:34.486479+0000","last_undegraded":"2024-03-11T21:59:34.486479+0000","last_fullsized":"2024-03-11T21:59:34.486479+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:22:50.747085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487485+0000","last_change":"2024-03-11T21:59:05.054646+0000","last_active":"2024-03-11T21:59:34.487485+0000","last_peered":"2024-03-11T21:59:34.487485+0000","last_clean":"2024-03-11T21:59:34.487485+0000","last_became_active":"2024-03-11T21:59:05.054042+0000","last_became_peered":"2024-03-11T21:59:05.054042+0000","last_unstale":"2024-03-11T21:59:34.487485+0000","last_undegraded":"2024-03-11T21:59:34.487485+0000","last_fullsized":"2024-03-11T21:59:34.487485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:53:27.471821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488779+0000","last_change":"2024-03-11T21:59:01.039314+0000","last_active":"2024-03-11T21:59:34.488779+0000","last_peered":"2024-03-11T21:59:34.488779+0000","last_clean":"2024-03-11T21:59:34.488779+0000","last_became_active":"2024-03-11T21:59:01.039031+0000","last_became_peered":"2024-03-11T21:59:01.039031+0000","last_unstale":"2024-03-11T21:59:34.488779+0000","last_undegraded":"2024-03-11T21:59:34.488779+0000","last_fullsized":"2024-03-11T21:59:34.488779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:56.584487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489116+0000","last_change":"2024-03-11T21:59:03.044410+0000","last_active":"2024-03-11T21:59:34.489116+0000","last_peered":"2024-03-11T21:59:34.489116+0000","last_clean":"2024-03-11T21:59:34.489116+0000","last_became_active":"2024-03-11T21:59:03.044107+0000","last_became_peered":"2024-03-11T21:59:03.044107+0000","last_unstale":"2024-03-11T21:59:34.489116+0000","last_undegraded":"2024-03-11T21:59:34.489116+0000","last_fullsized":"2024-03-11T21:59:34.489116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:51.957904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488129+0000","last_change":"2024-03-11T21:59:07.069069+0000","last_active":"2024-03-11T21:59:34.488129+0000","last_peered":"2024-03-11T21:59:34.488129+0000","last_clean":"2024-03-11T21:59:34.488129+0000","last_became_active":"2024-03-11T21:59:07.068774+0000","last_became_peered":"2024-03-11T21:59:07.068774+0000","last_unstale":"2024-03-11T21:59:34.488129+0000","last_undegraded":"2024-03-11T21:59:34.488129+0000","last_fullsized":"2024-03-11T21:59:34.488129+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:00:48.022324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489147+0000","last_change":"2024-03-11T21:59:05.054709+0000","last_active":"2024-03-11T21:59:34.489147+0000","last_peered":"2024-03-11T21:59:34.489147+0000","last_clean":"2024-03-11T21:59:34.489147+0000","last_became_active":"2024-03-11T21:59:05.053399+0000","last_became_peered":"2024-03-11T21:59:05.053399+0000","last_unstale":"2024-03-11T21:59:34.489147+0000","last_undegraded":"2024-03-11T21:59:34.489147+0000","last_fullsized":"2024-03-11T21:59:34.489147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:04.326875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489836+0000","last_change":"2024-03-11T21:59:01.039388+0000","last_active":"2024-03-11T21:59:34.489836+0000","last_peered":"2024-03-11T21:59:34.489836+0000","last_clean":"2024-03-11T21:59:34.489836+0000","last_became_active":"2024-03-11T21:59:01.038640+0000","last_became_peered":"2024-03-11T21:59:01.038640+0000","last_unstale":"2024-03-11T21:59:34.489836+0000","last_undegraded":"2024-03-11T21:59:34.489836+0000","last_fullsized":"2024-03-11T21:59:34.489836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:35:05.934298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488805+0000","last_change":"2024-03-11T21:59:03.041115+0000","last_active":"2024-03-11T21:59:34.488805+0000","last_peered":"2024-03-11T21:59:34.488805+0000","last_clean":"2024-03-11T21:59:34.488805+0000","last_became_active":"2024-03-11T21:59:03.040615+0000","last_became_peered":"2024-03-11T21:59:03.040615+0000","last_unstale":"2024-03-11T21:59:34.488805+0000","last_undegraded":"2024-03-11T21:59:34.488805+0000","last_fullsized":"2024-03-11T21:59:34.488805+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:29.602749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488330+0000","last_change":"2024-03-11T21:59:07.067463+0000","last_active":"2024-03-11T21:59:34.488330+0000","last_peered":"2024-03-11T21:59:34.488330+0000","last_clean":"2024-03-11T21:59:34.488330+0000","last_became_active":"2024-03-11T21:59:07.067122+0000","last_became_peered":"2024-03-11T21:59:07.067122+0000","last_unstale":"2024-03-11T21:59:34.488330+0000","last_undegraded":"2024-03-11T21:59:34.488330+0000","last_fullsized":"2024-03-11T21:59:34.488330+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:04.423427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488007+0000","last_change":"2024-03-11T21:59:05.054233+0000","last_active":"2024-03-11T21:59:34.488007+0000","last_peered":"2024-03-11T21:59:34.488007+0000","last_clean":"2024-03-11T21:59:34.488007+0000","last_became_active":"2024-03-11T21:59:05.052959+0000","last_became_peered":"2024-03-11T21:59:05.052959+0000","last_unstale":"2024-03-11T21:59:34.488007+0000","last_undegraded":"2024-03-11T21:59:34.488007+0000","last_fullsized":"2024-03-11T21:59:34.488007+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:24.691679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490831+0000","last_change":"2024-03-11T21:59:01.038993+0000","last_active":"2024-03-11T21:59:34.490831+0000","last_peered":"2024-03-11T21:59:34.490831+0000","last_clean":"2024-03-11T21:59:34.490831+0000","last_became_active":"2024-03-11T21:59:01.038611+0000","last_became_peered":"2024-03-11T21:59:01.038611+0000","last_unstale":"2024-03-11T21:59:34.490831+0000","last_undegraded":"2024-03-11T21:59:34.490831+0000","last_fullsized":"2024-03-11T21:59:34.490831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:38:01.953125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489621+0000","last_change":"2024-03-11T21:59:03.047975+0000","last_active":"2024-03-11T21:59:34.489621+0000","last_peered":"2024-03-11T21:59:34.489621+0000","last_clean":"2024-03-11T21:59:34.489621+0000","last_became_active":"2024-03-11T21:59:03.047684+0000","last_became_peered":"2024-03-11T21:59:03.047684+0000","last_unstale":"2024-03-11T21:59:34.489621+0000","last_undegraded":"2024-03-11T21:59:34.489621+0000","last_fullsized":"2024-03-11T21:59:34.489621+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:20.969101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488441+0000","last_change":"2024-03-11T21:59:07.069304+0000","last_active":"2024-03-11T21:59:34.488441+0000","last_peered":"2024-03-11T21:59:34.488441+0000","last_clean":"2024-03-11T21:59:34.488441+0000","last_became_active":"2024-03-11T21:59:07.068544+0000","last_became_peered":"2024-03-11T21:59:07.068544+0000","last_unstale":"2024-03-11T21:59:34.488441+0000","last_undegraded":"2024-03-11T21:59:34.488441+0000","last_fullsized":"2024-03-11T21:59:34.488441+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:47.017277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489011+0000","last_change":"2024-03-11T21:59:05.056872+0000","last_active":"2024-03-11T21:59:34.489011+0000","last_peered":"2024-03-11T21:59:34.489011+0000","last_clean":"2024-03-11T21:59:34.489011+0000","last_became_active":"2024-03-11T21:59:05.056591+0000","last_became_peered":"2024-03-11T21:59:05.056591+0000","last_unstale":"2024-03-11T21:59:34.489011+0000","last_undegraded":"2024-03-11T21:59:34.489011+0000","last_fullsized":"2024-03-11T21:59:34.489011+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:10.415552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490598+0000","last_change":"2024-03-11T21:59:01.036174+0000","last_active":"2024-03-11T21:59:34.490598+0000","last_peered":"2024-03-11T21:59:34.490598+0000","last_clean":"2024-03-11T21:59:34.490598+0000","last_became_active":"2024-03-11T21:59:01.035879+0000","last_became_peered":"2024-03-11T21:59:01.035879+0000","last_unstale":"2024-03-11T21:59:34.490598+0000","last_undegraded":"2024-03-11T21:59:34.490598+0000","last_fullsized":"2024-03-11T21:59:34.490598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:34.211387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489059+0000","last_change":"2024-03-11T21:59:03.046831+0000","last_active":"2024-03-11T21:59:34.489059+0000","last_peered":"2024-03-11T21:59:34.489059+0000","last_clean":"2024-03-11T21:59:34.489059+0000","last_became_active":"2024-03-11T21:59:03.046539+0000","last_became_peered":"2024-03-11T21:59:03.046539+0000","last_unstale":"2024-03-11T21:59:34.489059+0000","last_undegraded":"2024-03-11T21:59:34.489059+0000","last_fullsized":"2024-03-11T21:59:34.489059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:54.629296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487865+0000","last_change":"2024-03-11T21:59:07.077512+0000","last_active":"2024-03-11T21:59:34.487865+0000","last_peered":"2024-03-11T21:59:34.487865+0000","last_clean":"2024-03-11T21:59:34.487865+0000","last_became_active":"2024-03-11T21:59:07.077118+0000","last_became_peered":"2024-03-11T21:59:07.077118+0000","last_unstale":"2024-03-11T21:59:34.487865+0000","last_undegraded":"2024-03-11T21:59:34.487865+0000","last_fullsized":"2024-03-11T21:59:34.487865+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:22.796422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489421+0000","last_change":"2024-03-11T21:59:05.056343+0000","last_active":"2024-03-11T21:59:34.489421+0000","last_peered":"2024-03-11T21:59:34.489421+0000","last_clean":"2024-03-11T21:59:34.489421+0000","last_became_active":"2024-03-11T21:59:05.056003+0000","last_became_peered":"2024-03-11T21:59:05.056003+0000","last_unstale":"2024-03-11T21:59:34.489421+0000","last_undegraded":"2024-03-11T21:59:34.489421+0000","last_fullsized":"2024-03-11T21:59:34.489421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:06.300941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488318+0000","last_change":"2024-03-11T21:59:01.037389+0000","last_active":"2024-03-11T21:59:34.488318+0000","last_peered":"2024-03-11T21:59:34.488318+0000","last_clean":"2024-03-11T21:59:34.488318+0000","last_became_active":"2024-03-11T21:59:01.036598+0000","last_became_peered":"2024-03-11T21:59:01.036598+0000","last_unstale":"2024-03-11T21:59:34.488318+0000","last_undegraded":"2024-03-11T21:59:34.488318+0000","last_fullsized":"2024-03-11T21:59:34.488318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:02.005804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488847+0000","last_change":"2024-03-11T21:59:03.047801+0000","last_active":"2024-03-11T21:59:34.488847+0000","last_peered":"2024-03-11T21:59:34.488847+0000","last_clean":"2024-03-11T21:59:34.488847+0000","last_became_active":"2024-03-11T21:59:03.047493+0000","last_became_peered":"2024-03-11T21:59:03.047493+0000","last_unstale":"2024-03-11T21:59:34.488847+0000","last_undegraded":"2024-03-11T21:59:34.488847+0000","last_fullsized":"2024-03-11T21:59:34.488847+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:25.216397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487528+0000","last_change":"2024-03-11T21:59:07.077573+0000","last_active":"2024-03-11T21:59:34.487528+0000","last_peered":"2024-03-11T21:59:34.487528+0000","last_clean":"2024-03-11T21:59:34.487528+0000","last_became_active":"2024-03-11T21:59:07.077213+0000","last_became_peered":"2024-03-11T21:59:07.077213+0000","last_unstale":"2024-03-11T21:59:34.487528+0000","last_undegraded":"2024-03-11T21:59:34.487528+0000","last_fullsized":"2024-03-11T21:59:34.487528+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:47:00.146940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489163+0000","last_change":"2024-03-11T21:59:05.056491+0000","last_active":"2024-03-11T21:59:34.489163+0000","last_peered":"2024-03-11T21:59:34.489163+0000","last_clean":"2024-03-11T21:59:34.489163+0000","last_became_active":"2024-03-11T21:59:05.056186+0000","last_became_peered":"2024-03-11T21:59:05.056186+0000","last_unstale":"2024-03-11T21:59:34.489163+0000","last_undegraded":"2024-03-11T21:59:34.489163+0000","last_fullsized":"2024-03-11T21:59:34.489163+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:07:59.936686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486777+0000","last_change":"2024-03-11T21:59:01.039950+0000","last_active":"2024-03-11T21:59:34.486777+0000","last_peered":"2024-03-11T21:59:34.486777+0000","last_clean":"2024-03-11T21:59:34.486777+0000","last_became_active":"2024-03-11T21:59:01.039041+0000","last_became_peered":"2024-03-11T21:59:01.039041+0000","last_unstale":"2024-03-11T21:59:34.486777+0000","last_undegraded":"2024-03-11T21:59:34.486777+0000","last_fullsized":"2024-03-11T21:59:34.486777+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:44:12.294260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486204+0000","last_change":"2024-03-11T21:59:03.041612+0000","last_active":"2024-03-11T21:59:34.486204+0000","last_peered":"2024-03-11T21:59:34.486204+0000","last_clean":"2024-03-11T21:59:34.486204+0000","last_became_active":"2024-03-11T21:59:03.041309+0000","last_became_peered":"2024-03-11T21:59:03.041309+0000","last_unstale":"2024-03-11T21:59:34.486204+0000","last_undegraded":"2024-03-11T21:59:34.486204+0000","last_fullsized":"2024-03-11T21:59:34.486204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:52:18.163209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488987+0000","last_change":"2024-03-11T21:59:07.076528+0000","last_active":"2024-03-11T21:59:34.488987+0000","last_peered":"2024-03-11T21:59:34.488987+0000","last_clean":"2024-03-11T21:59:34.488987+0000","last_became_active":"2024-03-11T21:59:07.076163+0000","last_became_peered":"2024-03-11T21:59:07.076163+0000","last_unstale":"2024-03-11T21:59:34.488987+0000","last_undegraded":"2024-03-11T21:59:34.488987+0000","last_fullsized":"2024-03-11T21:59:34.488987+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:05:58.226928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487707+0000","last_change":"2024-03-11T21:59:05.054816+0000","last_active":"2024-03-11T21:59:34.487707+0000","last_peered":"2024-03-11T21:59:34.487707+0000","last_clean":"2024-03-11T21:59:34.487707+0000","last_became_active":"2024-03-11T21:59:05.053588+0000","last_became_peered":"2024-03-11T21:59:05.053588+0000","last_unstale":"2024-03-11T21:59:34.487707+0000","last_undegraded":"2024-03-11T21:59:34.487707+0000","last_fullsized":"2024-03-11T21:59:34.487707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:23:16.723768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488067+0000","last_change":"2024-03-11T21:59:01.040373+0000","last_active":"2024-03-11T21:59:34.488067+0000","last_peered":"2024-03-11T21:59:34.488067+0000","last_clean":"2024-03-11T21:59:34.488067+0000","last_became_active":"2024-03-11T21:59:01.040056+0000","last_became_peered":"2024-03-11T21:59:01.040056+0000","last_unstale":"2024-03-11T21:59:34.488067+0000","last_undegraded":"2024-03-11T21:59:34.488067+0000","last_fullsized":"2024-03-11T21:59:34.488067+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:03:47.842368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486455+0000","last_change":"2024-03-11T21:59:03.041541+0000","last_active":"2024-03-11T21:59:34.486455+0000","last_peered":"2024-03-11T21:59:34.486455+0000","last_clean":"2024-03-11T21:59:34.486455+0000","last_became_active":"2024-03-11T21:59:03.041266+0000","last_became_peered":"2024-03-11T21:59:03.041266+0000","last_unstale":"2024-03-11T21:59:34.486455+0000","last_undegraded":"2024-03-11T21:59:34.486455+0000","last_fullsized":"2024-03-11T21:59:34.486455+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.232039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487647+0000","last_change":"2024-03-11T21:59:07.076660+0000","last_active":"2024-03-11T21:59:34.487647+0000","last_peered":"2024-03-11T21:59:34.487647+0000","last_clean":"2024-03-11T21:59:34.487647+0000","last_became_active":"2024-03-11T21:59:07.076328+0000","last_became_peered":"2024-03-11T21:59:07.076328+0000","last_unstale":"2024-03-11T21:59:34.487647+0000","last_undegraded":"2024-03-11T21:59:34.487647+0000","last_fullsized":"2024-03-11T21:59:34.487647+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:30.305216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486375+0000","last_change":"2024-03-11T21:59:05.054252+0000","last_active":"2024-03-11T21:59:34.486375+0000","last_peered":"2024-03-11T21:59:34.486375+0000","last_clean":"2024-03-11T21:59:34.486375+0000","last_became_active":"2024-03-11T21:59:05.053418+0000","last_became_peered":"2024-03-11T21:59:05.053418+0000","last_unstale":"2024-03-11T21:59:34.486375+0000","last_undegraded":"2024-03-11T21:59:34.486375+0000","last_fullsized":"2024-03-11T21:59:34.486375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:00:26.879610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489767+0000","last_change":"2024-03-11T21:59:01.040450+0000","last_active":"2024-03-11T21:59:34.489767+0000","last_peered":"2024-03-11T21:59:34.489767+0000","last_clean":"2024-03-11T21:59:34.489767+0000","last_became_active":"2024-03-11T21:59:01.039996+0000","last_became_peered":"2024-03-11T21:59:01.039996+0000","last_unstale":"2024-03-11T21:59:34.489767+0000","last_undegraded":"2024-03-11T21:59:34.489767+0000","last_fullsized":"2024-03-11T21:59:34.489767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:10.643347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489965+0000","last_change":"2024-03-11T21:59:03.042457+0000","last_active":"2024-03-11T21:59:34.489965+0000","last_peered":"2024-03-11T21:59:34.489965+0000","last_clean":"2024-03-11T21:59:34.489965+0000","last_became_active":"2024-03-11T21:59:03.042105+0000","last_became_peered":"2024-03-11T21:59:03.042105+0000","last_unstale":"2024-03-11T21:59:34.489965+0000","last_undegraded":"2024-03-11T21:59:34.489965+0000","last_fullsized":"2024-03-11T21:59:34.489965+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:37.463970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487539+0000","last_change":"2024-03-11T21:59:07.067478+0000","last_active":"2024-03-11T21:59:34.487539+0000","last_peered":"2024-03-11T21:59:34.487539+0000","last_clean":"2024-03-11T21:59:34.487539+0000","last_became_active":"2024-03-11T21:59:07.067076+0000","last_became_peered":"2024-03-11T21:59:07.067076+0000","last_unstale":"2024-03-11T21:59:34.487539+0000","last_undegraded":"2024-03-11T21:59:34.487539+0000","last_fullsized":"2024-03-11T21:59:34.487539+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:04.533060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486522+0000","last_change":"2024-03-11T21:59:05.052745+0000","last_active":"2024-03-11T21:59:34.486522+0000","last_peered":"2024-03-11T21:59:34.486522+0000","last_clean":"2024-03-11T21:59:34.486522+0000","last_became_active":"2024-03-11T21:59:05.051792+0000","last_became_peered":"2024-03-11T21:59:05.051792+0000","last_unstale":"2024-03-11T21:59:34.486522+0000","last_undegraded":"2024-03-11T21:59:34.486522+0000","last_fullsized":"2024-03-11T21:59:34.486522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:00:08.381923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489647+0000","last_change":"2024-03-11T21:59:01.037649+0000","last_active":"2024-03-11T21:59:34.489647+0000","last_peered":"2024-03-11T21:59:34.489647+0000","last_clean":"2024-03-11T21:59:34.489647+0000","last_became_active":"2024-03-11T21:59:01.036686+0000","last_became_peered":"2024-03-11T21:59:01.036686+0000","last_unstale":"2024-03-11T21:59:34.489647+0000","last_undegraded":"2024-03-11T21:59:34.489647+0000","last_fullsized":"2024-03-11T21:59:34.489647+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:53.644624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489254+0000","last_change":"2024-03-11T21:59:03.044372+0000","last_active":"2024-03-11T21:59:34.489254+0000","last_peered":"2024-03-11T21:59:34.489254+0000","last_clean":"2024-03-11T21:59:34.489254+0000","last_became_active":"2024-03-11T21:59:03.044064+0000","last_became_peered":"2024-03-11T21:59:03.044064+0000","last_unstale":"2024-03-11T21:59:34.489254+0000","last_undegraded":"2024-03-11T21:59:34.489254+0000","last_fullsized":"2024-03-11T21:59:34.489254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:26:36.666112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488356+0000","last_change":"2024-03-11T21:59:07.069658+0000","last_active":"2024-03-11T21:59:34.488356+0000","last_peered":"2024-03-11T21:59:34.488356+0000","last_clean":"2024-03-11T21:59:34.488356+0000","last_became_active":"2024-03-11T21:59:07.069285+0000","last_became_peered":"2024-03-11T21:59:07.069285+0000","last_unstale":"2024-03-11T21:59:34.488356+0000","last_undegraded":"2024-03-11T21:59:34.488356+0000","last_fullsized":"2024-03-11T21:59:34.488356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:09.893670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488674+0000","last_change":"2024-03-11T21:59:05.054875+0000","last_active":"2024-03-11T21:59:34.488674+0000","last_peered":"2024-03-11T21:59:34.488674+0000","last_clean":"2024-03-11T21:59:34.488674+0000","last_became_active":"2024-03-11T21:59:05.054487+0000","last_became_peered":"2024-03-11T21:59:05.054487+0000","last_unstale":"2024-03-11T21:59:34.488674+0000","last_undegraded":"2024-03-11T21:59:34.488674+0000","last_fullsized":"2024-03-11T21:59:34.488674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:59.856730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489529+0000","last_change":"2024-03-11T21:59:01.039665+0000","last_active":"2024-03-11T21:59:34.489529+0000","last_peered":"2024-03-11T21:59:34.489529+0000","last_clean":"2024-03-11T21:59:34.489529+0000","last_became_active":"2024-03-11T21:59:01.039027+0000","last_became_peered":"2024-03-11T21:59:01.039027+0000","last_unstale":"2024-03-11T21:59:34.489529+0000","last_undegraded":"2024-03-11T21:59:34.489529+0000","last_fullsized":"2024-03-11T21:59:34.489529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:32.093622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489023+0000","last_change":"2024-03-11T21:59:03.042669+0000","last_active":"2024-03-11T21:59:34.489023+0000","last_peered":"2024-03-11T21:59:34.489023+0000","last_clean":"2024-03-11T21:59:34.489023+0000","last_became_active":"2024-03-11T21:59:03.041986+0000","last_became_peered":"2024-03-11T21:59:03.041986+0000","last_unstale":"2024-03-11T21:59:34.489023+0000","last_undegraded":"2024-03-11T21:59:34.489023+0000","last_fullsized":"2024-03-11T21:59:34.489023+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:10.531645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488176+0000","last_change":"2024-03-11T21:59:07.069211+0000","last_active":"2024-03-11T21:59:34.488176+0000","last_peered":"2024-03-11T21:59:34.488176+0000","last_clean":"2024-03-11T21:59:34.488176+0000","last_became_active":"2024-03-11T21:59:07.068840+0000","last_became_peered":"2024-03-11T21:59:07.068840+0000","last_unstale":"2024-03-11T21:59:34.488176+0000","last_undegraded":"2024-03-11T21:59:34.488176+0000","last_fullsized":"2024-03-11T21:59:34.488176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:57.300601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486660+0000","last_change":"2024-03-11T21:59:05.054401+0000","last_active":"2024-03-11T21:59:34.486660+0000","last_peered":"2024-03-11T21:59:34.486660+0000","last_clean":"2024-03-11T21:59:34.486660+0000","last_became_active":"2024-03-11T21:59:05.053501+0000","last_became_peered":"2024-03-11T21:59:05.053501+0000","last_unstale":"2024-03-11T21:59:34.486660+0000","last_undegraded":"2024-03-11T21:59:34.486660+0000","last_fullsized":"2024-03-11T21:59:34.486660+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:00:56.193895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488676+0000","last_change":"2024-03-11T21:59:01.036618+0000","last_active":"2024-03-11T21:59:34.488676+0000","last_peered":"2024-03-11T21:59:34.488676+0000","last_clean":"2024-03-11T21:59:34.488676+0000","last_became_active":"2024-03-11T21:59:01.036043+0000","last_became_peered":"2024-03-11T21:59:01.036043+0000","last_unstale":"2024-03-11T21:59:34.488676+0000","last_undegraded":"2024-03-11T21:59:34.488676+0000","last_fullsized":"2024-03-11T21:59:34.488676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:29:33.184690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489295+0000","last_change":"2024-03-11T21:59:03.046920+0000","last_active":"2024-03-11T21:59:34.489295+0000","last_peered":"2024-03-11T21:59:34.489295+0000","last_clean":"2024-03-11T21:59:34.489295+0000","last_became_active":"2024-03-11T21:59:03.046607+0000","last_became_peered":"2024-03-11T21:59:03.046607+0000","last_unstale":"2024-03-11T21:59:34.489295+0000","last_undegraded":"2024-03-11T21:59:34.489295+0000","last_fullsized":"2024-03-11T21:59:34.489295+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:37:18.267340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488221+0000","last_change":"2024-03-11T21:59:07.078856+0000","last_active":"2024-03-11T21:59:34.488221+0000","last_peered":"2024-03-11T21:59:34.488221+0000","last_clean":"2024-03-11T21:59:34.488221+0000","last_became_active":"2024-03-11T21:59:07.078264+0000","last_became_peered":"2024-03-11T21:59:07.078264+0000","last_unstale":"2024-03-11T21:59:34.488221+0000","last_undegraded":"2024-03-11T21:59:34.488221+0000","last_fullsized":"2024-03-11T21:59:34.488221+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:54:12.656535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489231+0000","last_change":"2024-03-11T21:59:05.056760+0000","last_active":"2024-03-11T21:59:34.489231+0000","last_peered":"2024-03-11T21:59:34.489231+0000","last_clean":"2024-03-11T21:59:34.489231+0000","last_became_active":"2024-03-11T21:59:05.056179+0000","last_became_peered":"2024-03-11T21:59:05.056179+0000","last_unstale":"2024-03-11T21:59:34.489231+0000","last_undegraded":"2024-03-11T21:59:34.489231+0000","last_fullsized":"2024-03-11T21:59:34.489231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:29:17.460397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488616+0000","last_change":"2024-03-11T21:59:01.036133+0000","last_active":"2024-03-11T21:59:34.488616+0000","last_peered":"2024-03-11T21:59:34.488616+0000","last_clean":"2024-03-11T21:59:34.488616+0000","last_became_active":"2024-03-11T21:59:01.035644+0000","last_became_peered":"2024-03-11T21:59:01.035644+0000","last_unstale":"2024-03-11T21:59:34.488616+0000","last_undegraded":"2024-03-11T21:59:34.488616+0000","last_fullsized":"2024-03-11T21:59:34.488616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:16:38.487046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486983+0000","last_change":"2024-03-11T21:59:03.040782+0000","last_active":"2024-03-11T21:59:34.486983+0000","last_peered":"2024-03-11T21:59:34.486983+0000","last_clean":"2024-03-11T21:59:34.486983+0000","last_became_active":"2024-03-11T21:59:03.040534+0000","last_became_peered":"2024-03-11T21:59:03.040534+0000","last_unstale":"2024-03-11T21:59:34.486983+0000","last_undegraded":"2024-03-11T21:59:34.486983+0000","last_fullsized":"2024-03-11T21:59:34.486983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:10:32.311227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488989+0000","last_change":"2024-03-11T21:59:07.067674+0000","last_active":"2024-03-11T21:59:34.488989+0000","last_peered":"2024-03-11T21:59:34.488989+0000","last_clean":"2024-03-11T21:59:34.488989+0000","last_became_active":"2024-03-11T21:59:07.067275+0000","last_became_peered":"2024-03-11T21:59:07.067275+0000","last_unstale":"2024-03-11T21:59:34.488989+0000","last_undegraded":"2024-03-11T21:59:34.488989+0000","last_fullsized":"2024-03-11T21:59:34.488989+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:51.192763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489473+0000","last_change":"2024-03-11T21:59:05.055282+0000","last_active":"2024-03-11T21:59:34.489473+0000","last_peered":"2024-03-11T21:59:34.489473+0000","last_clean":"2024-03-11T21:59:34.489473+0000","last_became_active":"2024-03-11T21:59:05.054882+0000","last_became_peered":"2024-03-11T21:59:05.054882+0000","last_unstale":"2024-03-11T21:59:34.489473+0000","last_undegraded":"2024-03-11T21:59:34.489473+0000","last_fullsized":"2024-03-11T21:59:34.489473+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:47:30.146828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488604+0000","last_change":"2024-03-11T21:59:01.038760+0000","last_active":"2024-03-11T21:59:34.488604+0000","last_peered":"2024-03-11T21:59:34.488604+0000","last_clean":"2024-03-11T21:59:34.488604+0000","last_became_active":"2024-03-11T21:59:01.038373+0000","last_became_peered":"2024-03-11T21:59:01.038373+0000","last_unstale":"2024-03-11T21:59:34.488604+0000","last_undegraded":"2024-03-11T21:59:34.488604+0000","last_fullsized":"2024-03-11T21:59:34.488604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:25.523518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488995+0000","last_change":"2024-03-11T21:59:03.045811+0000","last_active":"2024-03-11T21:59:34.488995+0000","last_peered":"2024-03-11T21:59:34.488995+0000","last_clean":"2024-03-11T21:59:34.488995+0000","last_became_active":"2024-03-11T21:59:03.045390+0000","last_became_peered":"2024-03-11T21:59:03.045390+0000","last_unstale":"2024-03-11T21:59:34.488995+0000","last_undegraded":"2024-03-11T21:59:34.488995+0000","last_fullsized":"2024-03-11T21:59:34.488995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:00.728036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489184+0000","last_change":"2024-03-11T21:59:07.069040+0000","last_active":"2024-03-11T21:59:34.489184+0000","last_peered":"2024-03-11T21:59:34.489184+0000","last_clean":"2024-03-11T21:59:34.489184+0000","last_became_active":"2024-03-11T21:59:07.068717+0000","last_became_peered":"2024-03-11T21:59:07.068717+0000","last_unstale":"2024-03-11T21:59:34.489184+0000","last_undegraded":"2024-03-11T21:59:34.489184+0000","last_fullsized":"2024-03-11T21:59:34.489184+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:48:25.192503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489030+0000","last_change":"2024-03-11T21:59:05.054655+0000","last_active":"2024-03-11T21:59:34.489030+0000","last_peered":"2024-03-11T21:59:34.489030+0000","last_clean":"2024-03-11T21:59:34.489030+0000","last_became_active":"2024-03-11T21:59:05.053654+0000","last_became_peered":"2024-03-11T21:59:05.053654+0000","last_unstale":"2024-03-11T21:59:34.489030+0000","last_undegraded":"2024-03-11T21:59:34.489030+0000","last_fullsized":"2024-03-11T21:59:34.489030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:16:32.862281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487652+0000","last_change":"2024-03-11T21:59:01.040250+0000","last_active":"2024-03-11T21:59:34.487652+0000","last_peered":"2024-03-11T21:59:34.487652+0000","last_clean":"2024-03-11T21:59:34.487652+0000","last_became_active":"2024-03-11T21:59:01.039866+0000","last_became_peered":"2024-03-11T21:59:01.039866+0000","last_unstale":"2024-03-11T21:59:34.487652+0000","last_undegraded":"2024-03-11T21:59:34.487652+0000","last_fullsized":"2024-03-11T21:59:34.487652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:29.952287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487223+0000","last_change":"2024-03-11T21:59:03.041103+0000","last_active":"2024-03-11T21:59:34.487223+0000","last_peered":"2024-03-11T21:59:34.487223+0000","last_clean":"2024-03-11T21:59:34.487223+0000","last_became_active":"2024-03-11T21:59:03.040742+0000","last_became_peered":"2024-03-11T21:59:03.040742+0000","last_unstale":"2024-03-11T21:59:34.487223+0000","last_undegraded":"2024-03-11T21:59:34.487223+0000","last_fullsized":"2024-03-11T21:59:34.487223+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:51:34.613463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487551+0000","last_change":"2024-03-11T21:59:07.077576+0000","last_active":"2024-03-11T21:59:34.487551+0000","last_peered":"2024-03-11T21:59:34.487551+0000","last_clean":"2024-03-11T21:59:34.487551+0000","last_became_active":"2024-03-11T21:59:07.077208+0000","last_became_peered":"2024-03-11T21:59:07.077208+0000","last_unstale":"2024-03-11T21:59:34.487551+0000","last_undegraded":"2024-03-11T21:59:34.487551+0000","last_fullsized":"2024-03-11T21:59:34.487551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:32.921859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487797+0000","last_change":"2024-03-11T21:59:05.055377+0000","last_active":"2024-03-11T21:59:34.487797+0000","last_peered":"2024-03-11T21:59:34.487797+0000","last_clean":"2024-03-11T21:59:34.487797+0000","last_became_active":"2024-03-11T21:59:05.054747+0000","last_became_peered":"2024-03-11T21:59:05.054747+0000","last_unstale":"2024-03-11T21:59:34.487797+0000","last_undegraded":"2024-03-11T21:59:34.487797+0000","last_fullsized":"2024-03-11T21:59:34.487797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:30:08.968548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488243+0000","last_change":"2024-03-11T21:59:01.040374+0000","last_active":"2024-03-11T21:59:34.488243+0000","last_peered":"2024-03-11T21:59:34.488243+0000","last_clean":"2024-03-11T21:59:34.488243+0000","last_became_active":"2024-03-11T21:59:01.040042+0000","last_became_peered":"2024-03-11T21:59:01.040042+0000","last_unstale":"2024-03-11T21:59:34.488243+0000","last_undegraded":"2024-03-11T21:59:34.488243+0000","last_fullsized":"2024-03-11T21:59:34.488243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:53.091994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490962+0000","last_change":"2024-03-11T21:59:03.046665+0000","last_active":"2024-03-11T21:59:34.490962+0000","last_peered":"2024-03-11T21:59:34.490962+0000","last_clean":"2024-03-11T21:59:34.490962+0000","last_became_active":"2024-03-11T21:59:03.046390+0000","last_became_peered":"2024-03-11T21:59:03.046390+0000","last_unstale":"2024-03-11T21:59:34.490962+0000","last_undegraded":"2024-03-11T21:59:34.490962+0000","last_fullsized":"2024-03-11T21:59:34.490962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:44:32.521080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489932+0000","last_change":"2024-03-11T21:59:07.067571+0000","last_active":"2024-03-11T21:59:34.489932+0000","last_peered":"2024-03-11T21:59:34.489932+0000","last_clean":"2024-03-11T21:59:34.489932+0000","last_became_active":"2024-03-11T21:59:07.067352+0000","last_became_peered":"2024-03-11T21:59:07.067352+0000","last_unstale":"2024-03-11T21:59:34.489932+0000","last_undegraded":"2024-03-11T21:59:34.489932+0000","last_fullsized":"2024-03-11T21:59:34.489932+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:08.429269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489479+0000","last_change":"2024-03-11T21:59:01.039099+0000","last_active":"2024-03-11T21:59:34.489479+0000","last_peered":"2024-03-11T21:59:34.489479+0000","last_clean":"2024-03-11T21:59:34.489479+0000","last_became_active":"2024-03-11T21:59:01.038710+0000","last_became_peered":"2024-03-11T21:59:01.038710+0000","last_unstale":"2024-03-11T21:59:34.489479+0000","last_undegraded":"2024-03-11T21:59:34.489479+0000","last_fullsized":"2024-03-11T21:59:34.489479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:56.988219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490766+0000","last_change":"2024-03-11T21:59:03.048105+0000","last_active":"2024-03-11T21:59:34.490766+0000","last_peered":"2024-03-11T21:59:34.490766+0000","last_clean":"2024-03-11T21:59:34.490766+0000","last_became_active":"2024-03-11T21:59:03.047806+0000","last_became_peered":"2024-03-11T21:59:03.047806+0000","last_unstale":"2024-03-11T21:59:34.490766+0000","last_undegraded":"2024-03-11T21:59:34.490766+0000","last_fullsized":"2024-03-11T21:59:34.490766+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:45.989046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488048+0000","last_change":"2024-03-11T21:59:05.056835+0000","last_active":"2024-03-11T21:59:34.488048+0000","last_peered":"2024-03-11T21:59:34.488048+0000","last_clean":"2024-03-11T21:59:34.488048+0000","last_became_active":"2024-03-11T21:59:05.056414+0000","last_became_peered":"2024-03-11T21:59:05.056414+0000","last_unstale":"2024-03-11T21:59:34.488048+0000","last_undegraded":"2024-03-11T21:59:34.488048+0000","last_fullsized":"2024-03-11T21:59:34.488048+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:10.862153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487211+0000","last_change":"2024-03-11T21:59:07.075162+0000","last_active":"2024-03-11T21:59:34.487211+0000","last_peered":"2024-03-11T21:59:34.487211+0000","last_clean":"2024-03-11T21:59:34.487211+0000","last_became_active":"2024-03-11T21:59:07.074789+0000","last_became_peered":"2024-03-11T21:59:07.074789+0000","last_unstale":"2024-03-11T21:59:34.487211+0000","last_undegraded":"2024-03-11T21:59:34.487211+0000","last_fullsized":"2024-03-11T21:59:34.487211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:46.090113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488662+0000","last_change":"2024-03-11T21:59:01.037870+0000","last_active":"2024-03-11T21:59:34.488662+0000","last_peered":"2024-03-11T21:59:34.488662+0000","last_clean":"2024-03-11T21:59:34.488662+0000","last_became_active":"2024-03-11T21:59:01.037513+0000","last_became_peered":"2024-03-11T21:59:01.037513+0000","last_unstale":"2024-03-11T21:59:34.488662+0000","last_undegraded":"2024-03-11T21:59:34.488662+0000","last_fullsized":"2024-03-11T21:59:34.488662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:09:02.516378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490544+0000","last_change":"2024-03-11T21:59:03.042464+0000","last_active":"2024-03-11T21:59:34.490544+0000","last_peered":"2024-03-11T21:59:34.490544+0000","last_clean":"2024-03-11T21:59:34.490544+0000","last_became_active":"2024-03-11T21:59:03.041749+0000","last_became_peered":"2024-03-11T21:59:03.041749+0000","last_unstale":"2024-03-11T21:59:34.490544+0000","last_undegraded":"2024-03-11T21:59:34.490544+0000","last_fullsized":"2024-03-11T21:59:34.490544+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:29:07.358472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488446+0000","last_change":"2024-03-11T21:59:05.055463+0000","last_active":"2024-03-11T21:59:34.488446+0000","last_peered":"2024-03-11T21:59:34.488446+0000","last_clean":"2024-03-11T21:59:34.488446+0000","last_became_active":"2024-03-11T21:59:05.054956+0000","last_became_peered":"2024-03-11T21:59:05.054956+0000","last_unstale":"2024-03-11T21:59:34.488446+0000","last_undegraded":"2024-03-11T21:59:34.488446+0000","last_fullsized":"2024-03-11T21:59:34.488446+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:18.678651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487322+0000","last_change":"2024-03-11T21:59:07.075208+0000","last_active":"2024-03-11T21:59:34.487322+0000","last_peered":"2024-03-11T21:59:34.487322+0000","last_clean":"2024-03-11T21:59:34.487322+0000","last_became_active":"2024-03-11T21:59:07.074858+0000","last_became_peered":"2024-03-11T21:59:07.074858+0000","last_unstale":"2024-03-11T21:59:34.487322+0000","last_undegraded":"2024-03-11T21:59:34.487322+0000","last_fullsized":"2024-03-11T21:59:34.487322+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:37.643897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488209+0000","last_change":"2024-03-11T21:59:01.038101+0000","last_active":"2024-03-11T21:59:34.488209+0000","last_peered":"2024-03-11T21:59:34.488209+0000","last_clean":"2024-03-11T21:59:34.488209+0000","last_became_active":"2024-03-11T21:59:01.037608+0000","last_became_peered":"2024-03-11T21:59:01.037608+0000","last_unstale":"2024-03-11T21:59:34.488209+0000","last_undegraded":"2024-03-11T21:59:34.488209+0000","last_fullsized":"2024-03-11T21:59:34.488209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:10.750688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489393+0000","last_change":"2024-03-11T21:59:03.046029+0000","last_active":"2024-03-11T21:59:34.489393+0000","last_peered":"2024-03-11T21:59:34.489393+0000","last_clean":"2024-03-11T21:59:34.489393+0000","last_became_active":"2024-03-11T21:59:03.045626+0000","last_became_peered":"2024-03-11T21:59:03.045626+0000","last_unstale":"2024-03-11T21:59:34.489393+0000","last_undegraded":"2024-03-11T21:59:34.489393+0000","last_fullsized":"2024-03-11T21:59:34.489393+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:07:49.827952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488379+0000","last_change":"2024-03-11T21:59:05.054927+0000","last_active":"2024-03-11T21:59:34.488379+0000","last_peered":"2024-03-11T21:59:34.488379+0000","last_clean":"2024-03-11T21:59:34.488379+0000","last_became_active":"2024-03-11T21:59:05.054387+0000","last_became_peered":"2024-03-11T21:59:05.054387+0000","last_unstale":"2024-03-11T21:59:34.488379+0000","last_undegraded":"2024-03-11T21:59:34.488379+0000","last_fullsized":"2024-03-11T21:59:34.488379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:15.472313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488896+0000","last_change":"2024-03-11T21:59:07.077996+0000","last_active":"2024-03-11T21:59:34.488896+0000","last_peered":"2024-03-11T21:59:34.488896+0000","last_clean":"2024-03-11T21:59:34.488896+0000","last_became_active":"2024-03-11T21:59:07.077608+0000","last_became_peered":"2024-03-11T21:59:07.077608+0000","last_unstale":"2024-03-11T21:59:34.488896+0000","last_undegraded":"2024-03-11T21:59:34.488896+0000","last_fullsized":"2024-03-11T21:59:34.488896+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:35.503888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487778+0000","last_change":"2024-03-11T21:59:03.047737+0000","last_active":"2024-03-11T21:59:34.487778+0000","last_peered":"2024-03-11T21:59:34.487778+0000","last_clean":"2024-03-11T21:59:34.487778+0000","last_became_active":"2024-03-11T21:59:03.047477+0000","last_became_peered":"2024-03-11T21:59:03.047477+0000","last_unstale":"2024-03-11T21:59:34.487778+0000","last_undegraded":"2024-03-11T21:59:34.487778+0000","last_fullsized":"2024-03-11T21:59:34.487778+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:40:52.913016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486667+0000","last_change":"2024-03-11T21:59:01.038820+0000","last_active":"2024-03-11T21:59:34.486667+0000","last_peered":"2024-03-11T21:59:34.486667+0000","last_clean":"2024-03-11T21:59:34.486667+0000","last_became_active":"2024-03-11T21:59:01.037953+0000","last_became_peered":"2024-03-11T21:59:01.037953+0000","last_unstale":"2024-03-11T21:59:34.486667+0000","last_undegraded":"2024-03-11T21:59:34.486667+0000","last_fullsized":"2024-03-11T21:59:34.486667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:05:05.696632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489149+0000","last_change":"2024-03-11T21:59:05.056499+0000","last_active":"2024-03-11T21:59:34.489149+0000","last_peered":"2024-03-11T21:59:34.489149+0000","last_clean":"2024-03-11T21:59:34.489149+0000","last_became_active":"2024-03-11T21:59:05.055049+0000","last_became_peered":"2024-03-11T21:59:05.055049+0000","last_unstale":"2024-03-11T21:59:34.489149+0000","last_undegraded":"2024-03-11T21:59:34.489149+0000","last_fullsized":"2024-03-11T21:59:34.489149+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:46.434743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488441+0000","last_change":"2024-03-11T21:59:07.069421+0000","last_active":"2024-03-11T21:59:34.488441+0000","last_peered":"2024-03-11T21:59:34.488441+0000","last_clean":"2024-03-11T21:59:34.488441+0000","last_became_active":"2024-03-11T21:59:07.068925+0000","last_became_peered":"2024-03-11T21:59:07.068925+0000","last_unstale":"2024-03-11T21:59:34.488441+0000","last_undegraded":"2024-03-11T21:59:34.488441+0000","last_fullsized":"2024-03-11T21:59:34.488441+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:55:51.345506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486761+0000","last_change":"2024-03-11T21:59:03.042114+0000","last_active":"2024-03-11T21:59:34.486761+0000","last_peered":"2024-03-11T21:59:34.486761+0000","last_clean":"2024-03-11T21:59:34.486761+0000","last_became_active":"2024-03-11T21:59:03.041739+0000","last_became_peered":"2024-03-11T21:59:03.041739+0000","last_unstale":"2024-03-11T21:59:34.486761+0000","last_undegraded":"2024-03-11T21:59:34.486761+0000","last_fullsized":"2024-03-11T21:59:34.486761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:24.738017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489124+0000","last_change":"2024-03-11T21:59:01.041002+0000","last_active":"2024-03-11T21:59:34.489124+0000","last_peered":"2024-03-11T21:59:34.489124+0000","last_clean":"2024-03-11T21:59:34.489124+0000","last_became_active":"2024-03-11T21:59:01.040553+0000","last_became_peered":"2024-03-11T21:59:01.040553+0000","last_unstale":"2024-03-11T21:59:34.489124+0000","last_undegraded":"2024-03-11T21:59:34.489124+0000","last_fullsized":"2024-03-11T21:59:34.489124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:42:44.108320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.591324+0000","last_change":"2024-03-11T21:58:46.478599+0000","last_active":"2024-03-11T21:59:36.591324+0000","last_peered":"2024-03-11T21:59:36.591324+0000","last_clean":"2024-03-11T21:59:36.591324+0000","last_became_active":"2024-03-11T21:58:46.478232+0000","last_became_peered":"2024-03-11T21:58:46.478232+0000","last_unstale":"2024-03-11T21:59:36.591324+0000","last_undegraded":"2024-03-11T21:59:36.591324+0000","last_fullsized":"2024-03-11T21:59:36.591324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:58:45.465824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:58:45.465824+0000","last_clean_scrub_stamp":"2024-03-11T21:58:45.465824+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:45.199571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489745+0000","last_change":"2024-03-11T21:59:05.054324+0000","last_active":"2024-03-11T21:59:34.489745+0000","last_peered":"2024-03-11T21:59:34.489745+0000","last_clean":"2024-03-11T21:59:34.489745+0000","last_became_active":"2024-03-11T21:59:05.053974+0000","last_became_peered":"2024-03-11T21:59:05.053974+0000","last_unstale":"2024-03-11T21:59:34.489745+0000","last_undegraded":"2024-03-11T21:59:34.489745+0000","last_fullsized":"2024-03-11T21:59:34.489745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:06.466929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489253+0000","last_change":"2024-03-11T21:59:07.069411+0000","last_active":"2024-03-11T21:59:34.489253+0000","last_peered":"2024-03-11T21:59:34.489253+0000","last_clean":"2024-03-11T21:59:34.489253+0000","last_became_active":"2024-03-11T21:59:07.068720+0000","last_became_peered":"2024-03-11T21:59:07.068720+0000","last_unstale":"2024-03-11T21:59:34.489253+0000","last_undegraded":"2024-03-11T21:59:34.489253+0000","last_fullsized":"2024-03-11T21:59:34.489253+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:04:24.994863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486797+0000","last_change":"2024-03-11T21:59:03.040813+0000","last_active":"2024-03-11T21:59:34.486797+0000","last_peered":"2024-03-11T21:59:34.486797+0000","last_clean":"2024-03-11T21:59:34.486797+0000","last_became_active":"2024-03-11T21:59:03.040548+0000","last_became_peered":"2024-03-11T21:59:03.040548+0000","last_unstale":"2024-03-11T21:59:34.486797+0000","last_undegraded":"2024-03-11T21:59:34.486797+0000","last_fullsized":"2024-03-11T21:59:34.486797+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:14:27.751072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487230+0000","last_change":"2024-03-11T21:59:01.041126+0000","last_active":"2024-03-11T21:59:34.487230+0000","last_peered":"2024-03-11T21:59:34.487230+0000","last_clean":"2024-03-11T21:59:34.487230+0000","last_became_active":"2024-03-11T21:59:01.040771+0000","last_became_peered":"2024-03-11T21:59:01.040771+0000","last_unstale":"2024-03-11T21:59:34.487230+0000","last_undegraded":"2024-03-11T21:59:34.487230+0000","last_fullsized":"2024-03-11T21:59:34.487230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:08:13.551981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489736+0000","last_change":"2024-03-11T21:59:05.056172+0000","last_active":"2024-03-11T21:59:34.489736+0000","last_peered":"2024-03-11T21:59:34.489736+0000","last_clean":"2024-03-11T21:59:34.489736+0000","last_became_active":"2024-03-11T21:59:05.055820+0000","last_became_peered":"2024-03-11T21:59:05.055820+0000","last_unstale":"2024-03-11T21:59:34.489736+0000","last_undegraded":"2024-03-11T21:59:34.489736+0000","last_fullsized":"2024-03-11T21:59:34.489736+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:36.140674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488209+0000","last_change":"2024-03-11T21:59:07.069036+0000","last_active":"2024-03-11T21:59:34.488209+0000","last_peered":"2024-03-11T21:59:34.488209+0000","last_clean":"2024-03-11T21:59:34.488209+0000","last_became_active":"2024-03-11T21:59:07.068719+0000","last_became_peered":"2024-03-11T21:59:07.068719+0000","last_unstale":"2024-03-11T21:59:34.488209+0000","last_undegraded":"2024-03-11T21:59:34.488209+0000","last_fullsized":"2024-03-11T21:59:34.488209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:38:53.744781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489364+0000","last_change":"2024-03-11T21:59:01.038717+0000","last_active":"2024-03-11T21:59:34.489364+0000","last_peered":"2024-03-11T21:59:34.489364+0000","last_clean":"2024-03-11T21:59:34.489364+0000","last_became_active":"2024-03-11T21:59:01.038223+0000","last_became_peered":"2024-03-11T21:59:01.038223+0000","last_unstale":"2024-03-11T21:59:34.489364+0000","last_undegraded":"2024-03-11T21:59:34.489364+0000","last_fullsized":"2024-03-11T21:59:34.489364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:24.938122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488557+0000","last_change":"2024-03-11T21:59:03.042857+0000","last_active":"2024-03-11T21:59:34.488557+0000","last_peered":"2024-03-11T21:59:34.488557+0000","last_clean":"2024-03-11T21:59:34.488557+0000","last_became_active":"2024-03-11T21:59:03.042552+0000","last_became_peered":"2024-03-11T21:59:03.042552+0000","last_unstale":"2024-03-11T21:59:34.488557+0000","last_undegraded":"2024-03-11T21:59:34.488557+0000","last_fullsized":"2024-03-11T21:59:34.488557+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:14:27.133202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488775+0000","last_change":"2024-03-11T21:59:05.056668+0000","last_active":"2024-03-11T21:59:34.488775+0000","last_peered":"2024-03-11T21:59:34.488775+0000","last_clean":"2024-03-11T21:59:34.488775+0000","last_became_active":"2024-03-11T21:59:05.056359+0000","last_became_peered":"2024-03-11T21:59:05.056359+0000","last_unstale":"2024-03-11T21:59:34.488775+0000","last_undegraded":"2024-03-11T21:59:34.488775+0000","last_fullsized":"2024-03-11T21:59:34.488775+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:43.694022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487333+0000","last_change":"2024-03-11T21:59:07.067965+0000","last_active":"2024-03-11T21:59:34.487333+0000","last_peered":"2024-03-11T21:59:34.487333+0000","last_clean":"2024-03-11T21:59:34.487333+0000","last_became_active":"2024-03-11T21:59:07.067509+0000","last_became_peered":"2024-03-11T21:59:07.067509+0000","last_unstale":"2024-03-11T21:59:34.487333+0000","last_undegraded":"2024-03-11T21:59:34.487333+0000","last_fullsized":"2024-03-11T21:59:34.487333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:32:35.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488875+0000","last_change":"2024-03-11T21:59:01.038781+0000","last_active":"2024-03-11T21:59:34.488875+0000","last_peered":"2024-03-11T21:59:34.488875+0000","last_clean":"2024-03-11T21:59:34.488875+0000","last_became_active":"2024-03-11T21:59:01.037750+0000","last_became_peered":"2024-03-11T21:59:01.037750+0000","last_unstale":"2024-03-11T21:59:34.488875+0000","last_undegraded":"2024-03-11T21:59:34.488875+0000","last_fullsized":"2024-03-11T21:59:34.488875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:19.762311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489496+0000","last_change":"2024-03-11T21:59:03.042122+0000","last_active":"2024-03-11T21:59:34.489496+0000","last_peered":"2024-03-11T21:59:34.489496+0000","last_clean":"2024-03-11T21:59:34.489496+0000","last_became_active":"2024-03-11T21:59:03.041816+0000","last_became_peered":"2024-03-11T21:59:03.041816+0000","last_unstale":"2024-03-11T21:59:34.489496+0000","last_undegraded":"2024-03-11T21:59:34.489496+0000","last_fullsized":"2024-03-11T21:59:34.489496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:30:05.746944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486647+0000","last_change":"2024-03-11T21:59:05.052923+0000","last_active":"2024-03-11T21:59:34.486647+0000","last_peered":"2024-03-11T21:59:34.486647+0000","last_clean":"2024-03-11T21:59:34.486647+0000","last_became_active":"2024-03-11T21:59:05.052521+0000","last_became_peered":"2024-03-11T21:59:05.052521+0000","last_unstale":"2024-03-11T21:59:34.486647+0000","last_undegraded":"2024-03-11T21:59:34.486647+0000","last_fullsized":"2024-03-11T21:59:34.486647+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:36:41.641848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489930+0000","last_change":"2024-03-11T21:59:07.075344+0000","last_active":"2024-03-11T21:59:34.489930+0000","last_peered":"2024-03-11T21:59:34.489930+0000","last_clean":"2024-03-11T21:59:34.489930+0000","last_became_active":"2024-03-11T21:59:07.075082+0000","last_became_peered":"2024-03-11T21:59:07.075082+0000","last_unstale":"2024-03-11T21:59:34.489930+0000","last_undegraded":"2024-03-11T21:59:34.489930+0000","last_fullsized":"2024-03-11T21:59:34.489930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:18.502772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489540+0000","last_change":"2024-03-11T21:59:01.037903+0000","last_active":"2024-03-11T21:59:34.489540+0000","last_peered":"2024-03-11T21:59:34.489540+0000","last_clean":"2024-03-11T21:59:34.489540+0000","last_became_active":"2024-03-11T21:59:01.036809+0000","last_became_peered":"2024-03-11T21:59:01.036809+0000","last_unstale":"2024-03-11T21:59:34.489540+0000","last_undegraded":"2024-03-11T21:59:34.489540+0000","last_fullsized":"2024-03-11T21:59:34.489540+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:37:15.249603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487956+0000","last_change":"2024-03-11T21:59:03.043631+0000","last_active":"2024-03-11T21:59:34.487956+0000","last_peered":"2024-03-11T21:59:34.487956+0000","last_clean":"2024-03-11T21:59:34.487956+0000","last_became_active":"2024-03-11T21:59:03.043341+0000","last_became_peered":"2024-03-11T21:59:03.043341+0000","last_unstale":"2024-03-11T21:59:34.487956+0000","last_undegraded":"2024-03-11T21:59:34.487956+0000","last_fullsized":"2024-03-11T21:59:34.487956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:57.638571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488035+0000","last_change":"2024-03-11T21:59:05.056062+0000","last_active":"2024-03-11T21:59:34.488035+0000","last_peered":"2024-03-11T21:59:34.488035+0000","last_clean":"2024-03-11T21:59:34.488035+0000","last_became_active":"2024-03-11T21:59:05.055570+0000","last_became_peered":"2024-03-11T21:59:05.055570+0000","last_unstale":"2024-03-11T21:59:34.488035+0000","last_undegraded":"2024-03-11T21:59:34.488035+0000","last_fullsized":"2024-03-11T21:59:34.488035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:19:33.814384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489093+0000","last_change":"2024-03-11T21:59:07.068996+0000","last_active":"2024-03-11T21:59:34.489093+0000","last_peered":"2024-03-11T21:59:34.489093+0000","last_clean":"2024-03-11T21:59:34.489093+0000","last_became_active":"2024-03-11T21:59:07.068652+0000","last_became_peered":"2024-03-11T21:59:07.068652+0000","last_unstale":"2024-03-11T21:59:34.489093+0000","last_undegraded":"2024-03-11T21:59:34.489093+0000","last_fullsized":"2024-03-11T21:59:34.489093+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:16:25.487141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489490+0000","last_change":"2024-03-11T21:59:05.054058+0000","last_active":"2024-03-11T21:59:34.489490+0000","last_peered":"2024-03-11T21:59:34.489490+0000","last_clean":"2024-03-11T21:59:34.489490+0000","last_became_active":"2024-03-11T21:59:05.052901+0000","last_became_peered":"2024-03-11T21:59:05.052901+0000","last_unstale":"2024-03-11T21:59:34.489490+0000","last_undegraded":"2024-03-11T21:59:34.489490+0000","last_fullsized":"2024-03-11T21:59:34.489490+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:38.321381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487326+0000","last_change":"2024-03-11T21:59:01.041311+0000","last_active":"2024-03-11T21:59:34.487326+0000","last_peered":"2024-03-11T21:59:34.487326+0000","last_clean":"2024-03-11T21:59:34.487326+0000","last_became_active":"2024-03-11T21:59:01.040878+0000","last_became_peered":"2024-03-11T21:59:01.040878+0000","last_unstale":"2024-03-11T21:59:34.487326+0000","last_undegraded":"2024-03-11T21:59:34.487326+0000","last_fullsized":"2024-03-11T21:59:34.487326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:12.379926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490782+0000","last_change":"2024-03-11T21:59:03.045627+0000","last_active":"2024-03-11T21:59:34.490782+0000","last_peered":"2024-03-11T21:59:34.490782+0000","last_clean":"2024-03-11T21:59:34.490782+0000","last_became_active":"2024-03-11T21:59:03.045319+0000","last_became_peered":"2024-03-11T21:59:03.045319+0000","last_unstale":"2024-03-11T21:59:34.490782+0000","last_undegraded":"2024-03-11T21:59:34.490782+0000","last_fullsized":"2024-03-11T21:59:34.490782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:09.069520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489916+0000","last_change":"2024-03-11T21:59:07.069361+0000","last_active":"2024-03-11T21:59:34.489916+0000","last_peered":"2024-03-11T21:59:34.489916+0000","last_clean":"2024-03-11T21:59:34.489916+0000","last_became_active":"2024-03-11T21:59:07.068827+0000","last_became_peered":"2024-03-11T21:59:07.068827+0000","last_unstale":"2024-03-11T21:59:34.489916+0000","last_undegraded":"2024-03-11T21:59:34.489916+0000","last_fullsized":"2024-03-11T21:59:34.489916+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:07.505663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489066+0000","last_change":"2024-03-11T21:59:05.055222+0000","last_active":"2024-03-11T21:59:34.489066+0000","last_peered":"2024-03-11T21:59:34.489066+0000","last_clean":"2024-03-11T21:59:34.489066+0000","last_became_active":"2024-03-11T21:59:05.054668+0000","last_became_peered":"2024-03-11T21:59:05.054668+0000","last_unstale":"2024-03-11T21:59:34.489066+0000","last_undegraded":"2024-03-11T21:59:34.489066+0000","last_fullsized":"2024-03-11T21:59:34.489066+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:46:46.679656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486593+0000","last_change":"2024-03-11T21:59:01.039937+0000","last_active":"2024-03-11T21:59:34.486593+0000","last_peered":"2024-03-11T21:59:34.486593+0000","last_clean":"2024-03-11T21:59:34.486593+0000","last_became_active":"2024-03-11T21:59:01.039277+0000","last_became_peered":"2024-03-11T21:59:01.039277+0000","last_unstale":"2024-03-11T21:59:34.486593+0000","last_undegraded":"2024-03-11T21:59:34.486593+0000","last_fullsized":"2024-03-11T21:59:34.486593+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:05:27.755652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489534+0000","last_change":"2024-03-11T21:59:03.046380+0000","last_active":"2024-03-11T21:59:34.489534+0000","last_peered":"2024-03-11T21:59:34.489534+0000","last_clean":"2024-03-11T21:59:34.489534+0000","last_became_active":"2024-03-11T21:59:03.046051+0000","last_became_peered":"2024-03-11T21:59:03.046051+0000","last_unstale":"2024-03-11T21:59:34.489534+0000","last_undegraded":"2024-03-11T21:59:34.489534+0000","last_fullsized":"2024-03-11T21:59:34.489534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:08.018522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488908+0000","last_change":"2024-03-11T21:59:07.065874+0000","last_active":"2024-03-11T21:59:34.488908+0000","last_peered":"2024-03-11T21:59:34.488908+0000","last_clean":"2024-03-11T21:59:34.488908+0000","last_became_active":"2024-03-11T21:59:07.065633+0000","last_became_peered":"2024-03-11T21:59:07.065633+0000","last_unstale":"2024-03-11T21:59:34.488908+0000","last_undegraded":"2024-03-11T21:59:34.488908+0000","last_fullsized":"2024-03-11T21:59:34.488908+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:05:08.990245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489051+0000","last_change":"2024-03-11T21:59:01.035837+0000","last_active":"2024-03-11T21:59:34.489051+0000","last_peered":"2024-03-11T21:59:34.489051+0000","last_clean":"2024-03-11T21:59:34.489051+0000","last_became_active":"2024-03-11T21:59:01.035535+0000","last_became_peered":"2024-03-11T21:59:01.035535+0000","last_unstale":"2024-03-11T21:59:34.489051+0000","last_undegraded":"2024-03-11T21:59:34.489051+0000","last_fullsized":"2024-03-11T21:59:34.489051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:41.324819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488165+0000","last_change":"2024-03-11T21:59:03.045158+0000","last_active":"2024-03-11T21:59:34.488165+0000","last_peered":"2024-03-11T21:59:34.488165+0000","last_clean":"2024-03-11T21:59:34.488165+0000","last_became_active":"2024-03-11T21:59:03.044683+0000","last_became_peered":"2024-03-11T21:59:03.044683+0000","last_unstale":"2024-03-11T21:59:34.488165+0000","last_undegraded":"2024-03-11T21:59:34.488165+0000","last_fullsized":"2024-03-11T21:59:34.488165+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:52:39.490851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488562+0000","last_change":"2024-03-11T21:59:05.052899+0000","last_active":"2024-03-11T21:59:34.488562+0000","last_peered":"2024-03-11T21:59:34.488562+0000","last_clean":"2024-03-11T21:59:34.488562+0000","last_became_active":"2024-03-11T21:59:05.052542+0000","last_became_peered":"2024-03-11T21:59:05.052542+0000","last_unstale":"2024-03-11T21:59:34.488562+0000","last_undegraded":"2024-03-11T21:59:34.488562+0000","last_fullsized":"2024-03-11T21:59:34.488562+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:42.009843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487184+0000","last_change":"2024-03-11T21:59:07.078406+0000","last_active":"2024-03-11T21:59:34.487184+0000","last_peered":"2024-03-11T21:59:34.487184+0000","last_clean":"2024-03-11T21:59:34.487184+0000","last_became_active":"2024-03-11T21:59:07.078044+0000","last_became_peered":"2024-03-11T21:59:07.078044+0000","last_unstale":"2024-03-11T21:59:34.487184+0000","last_undegraded":"2024-03-11T21:59:34.487184+0000","last_fullsized":"2024-03-11T21:59:34.487184+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:27.192333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489683+0000","last_change":"2024-03-11T21:59:05.054810+0000","last_active":"2024-03-11T21:59:34.489683+0000","last_peered":"2024-03-11T21:59:34.489683+0000","last_clean":"2024-03-11T21:59:34.489683+0000","last_became_active":"2024-03-11T21:59:05.054482+0000","last_became_peered":"2024-03-11T21:59:05.054482+0000","last_unstale":"2024-03-11T21:59:34.489683+0000","last_undegraded":"2024-03-11T21:59:34.489683+0000","last_fullsized":"2024-03-11T21:59:34.489683+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:57:26.178250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488958+0000","last_change":"2024-03-11T21:59:01.035916+0000","last_active":"2024-03-11T21:59:34.488958+0000","last_peered":"2024-03-11T21:59:34.488958+0000","last_clean":"2024-03-11T21:59:34.488958+0000","last_became_active":"2024-03-11T21:59:01.035489+0000","last_became_peered":"2024-03-11T21:59:01.035489+0000","last_unstale":"2024-03-11T21:59:34.488958+0000","last_undegraded":"2024-03-11T21:59:34.488958+0000","last_fullsized":"2024-03-11T21:59:34.488958+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:46.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488387+0000","last_change":"2024-03-11T21:59:03.042161+0000","last_active":"2024-03-11T21:59:34.488387+0000","last_peered":"2024-03-11T21:59:34.488387+0000","last_clean":"2024-03-11T21:59:34.488387+0000","last_became_active":"2024-03-11T21:59:03.041755+0000","last_became_peered":"2024-03-11T21:59:03.041755+0000","last_unstale":"2024-03-11T21:59:34.488387+0000","last_undegraded":"2024-03-11T21:59:34.488387+0000","last_fullsized":"2024-03-11T21:59:34.488387+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:25.127890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488395+0000","last_change":"2024-03-11T21:59:07.067618+0000","last_active":"2024-03-11T21:59:34.488395+0000","last_peered":"2024-03-11T21:59:34.488395+0000","last_clean":"2024-03-11T21:59:34.488395+0000","last_became_active":"2024-03-11T21:59:07.067318+0000","last_became_peered":"2024-03-11T21:59:07.067318+0000","last_unstale":"2024-03-11T21:59:34.488395+0000","last_undegraded":"2024-03-11T21:59:34.488395+0000","last_fullsized":"2024-03-11T21:59:34.488395+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:44.518080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.491104+0000","last_change":"2024-03-11T21:59:01.039025+0000","last_active":"2024-03-11T21:59:34.491104+0000","last_peered":"2024-03-11T21:59:34.491104+0000","last_clean":"2024-03-11T21:59:34.491104+0000","last_became_active":"2024-03-11T21:59:01.038681+0000","last_became_peered":"2024-03-11T21:59:01.038681+0000","last_unstale":"2024-03-11T21:59:34.491104+0000","last_undegraded":"2024-03-11T21:59:34.491104+0000","last_fullsized":"2024-03-11T21:59:34.491104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:00.822840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490013+0000","last_change":"2024-03-11T21:59:03.042063+0000","last_active":"2024-03-11T21:59:34.490013+0000","last_peered":"2024-03-11T21:59:34.490013+0000","last_clean":"2024-03-11T21:59:34.490013+0000","last_became_active":"2024-03-11T21:59:03.041612+0000","last_became_peered":"2024-03-11T21:59:03.041612+0000","last_unstale":"2024-03-11T21:59:34.490013+0000","last_undegraded":"2024-03-11T21:59:34.490013+0000","last_fullsized":"2024-03-11T21:59:34.490013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:55.201857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488088+0000","last_change":"2024-03-11T21:59:05.055023+0000","last_active":"2024-03-11T21:59:34.488088+0000","last_peered":"2024-03-11T21:59:34.488088+0000","last_clean":"2024-03-11T21:59:34.488088+0000","last_became_active":"2024-03-11T21:59:05.054466+0000","last_became_peered":"2024-03-11T21:59:05.054466+0000","last_unstale":"2024-03-11T21:59:34.488088+0000","last_undegraded":"2024-03-11T21:59:34.488088+0000","last_fullsized":"2024-03-11T21:59:34.488088+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:26:08.514384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486366+0000","last_change":"2024-03-11T21:59:07.077441+0000","last_active":"2024-03-11T21:59:34.486366+0000","last_peered":"2024-03-11T21:59:34.486366+0000","last_clean":"2024-03-11T21:59:34.486366+0000","last_became_active":"2024-03-11T21:59:07.077036+0000","last_became_peered":"2024-03-11T21:59:07.077036+0000","last_unstale":"2024-03-11T21:59:34.486366+0000","last_undegraded":"2024-03-11T21:59:34.486366+0000","last_fullsized":"2024-03-11T21:59:34.486366+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:33:20.121043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489222+0000","last_change":"2024-03-11T21:59:01.038981+0000","last_active":"2024-03-11T21:59:34.489222+0000","last_peered":"2024-03-11T21:59:34.489222+0000","last_clean":"2024-03-11T21:59:34.489222+0000","last_became_active":"2024-03-11T21:59:01.038414+0000","last_became_peered":"2024-03-11T21:59:01.038414+0000","last_unstale":"2024-03-11T21:59:34.489222+0000","last_undegraded":"2024-03-11T21:59:34.489222+0000","last_fullsized":"2024-03-11T21:59:34.489222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:24.519203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488696+0000","last_change":"2024-03-11T21:59:03.042154+0000","last_active":"2024-03-11T21:59:34.488696+0000","last_peered":"2024-03-11T21:59:34.488696+0000","last_clean":"2024-03-11T21:59:34.488696+0000","last_became_active":"2024-03-11T21:59:03.041645+0000","last_became_peered":"2024-03-11T21:59:03.041645+0000","last_unstale":"2024-03-11T21:59:34.488696+0000","last_undegraded":"2024-03-11T21:59:34.488696+0000","last_fullsized":"2024-03-11T21:59:34.488696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:12:26.101680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487378+0000","last_change":"2024-03-11T21:59:05.054428+0000","last_active":"2024-03-11T21:59:34.487378+0000","last_peered":"2024-03-11T21:59:34.487378+0000","last_clean":"2024-03-11T21:59:34.487378+0000","last_became_active":"2024-03-11T21:59:05.053502+0000","last_became_peered":"2024-03-11T21:59:05.053502+0000","last_unstale":"2024-03-11T21:59:34.487378+0000","last_undegraded":"2024-03-11T21:59:34.487378+0000","last_fullsized":"2024-03-11T21:59:34.487378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:19:55.095498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487405+0000","last_change":"2024-03-11T21:59:07.067728+0000","last_active":"2024-03-11T21:59:34.487405+0000","last_peered":"2024-03-11T21:59:34.487405+0000","last_clean":"2024-03-11T21:59:34.487405+0000","last_became_active":"2024-03-11T21:59:07.067145+0000","last_became_peered":"2024-03-11T21:59:07.067145+0000","last_unstale":"2024-03-11T21:59:34.487405+0000","last_undegraded":"2024-03-11T21:59:34.487405+0000","last_fullsized":"2024-03-11T21:59:34.487405+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:08.065774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488232+0000","last_change":"2024-03-11T21:59:01.037676+0000","last_active":"2024-03-11T21:59:34.488232+0000","last_peered":"2024-03-11T21:59:34.488232+0000","last_clean":"2024-03-11T21:59:34.488232+0000","last_became_active":"2024-03-11T21:59:01.037380+0000","last_became_peered":"2024-03-11T21:59:01.037380+0000","last_unstale":"2024-03-11T21:59:34.488232+0000","last_undegraded":"2024-03-11T21:59:34.488232+0000","last_fullsized":"2024-03-11T21:59:34.488232+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:17.909275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489646+0000","last_change":"2024-03-11T21:59:03.042871+0000","last_active":"2024-03-11T21:59:34.489646+0000","last_peered":"2024-03-11T21:59:34.489646+0000","last_clean":"2024-03-11T21:59:34.489646+0000","last_became_active":"2024-03-11T21:59:03.042446+0000","last_became_peered":"2024-03-11T21:59:03.042446+0000","last_unstale":"2024-03-11T21:59:34.489646+0000","last_undegraded":"2024-03-11T21:59:34.489646+0000","last_fullsized":"2024-03-11T21:59:34.489646+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:26:04.488628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489751+0000","last_change":"2024-03-11T21:59:05.051540+0000","last_active":"2024-03-11T21:59:34.489751+0000","last_peered":"2024-03-11T21:59:34.489751+0000","last_clean":"2024-03-11T21:59:34.489751+0000","last_became_active":"2024-03-11T21:59:05.051219+0000","last_became_peered":"2024-03-11T21:59:05.051219+0000","last_unstale":"2024-03-11T21:59:34.489751+0000","last_undegraded":"2024-03-11T21:59:34.489751+0000","last_fullsized":"2024-03-11T21:59:34.489751+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:27.734526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738393,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:59:44.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:59:45.148 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:46.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:46 smithi151 ceph-mon[25773]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:59:46.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:46 smithi176 ceph-mon[27622]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:59:46.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:46 smithi189 ceph-mon[27727]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:59:48.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:48 smithi151 ceph-mon[25773]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:59:48.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:48 smithi176 ceph-mon[27622]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:59:48.614 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:59:48.616 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-03-11T21:59:48.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:48 smithi189 ceph-mon[27727]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:59:49.994 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-11T21:59:48.498712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002521"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487575+0000","last_change":"2024-03-11T21:59:07.075476+0000","last_active":"2024-03-11T21:59:34.487575+0000","last_peered":"2024-03-11T21:59:34.487575+0000","last_clean":"2024-03-11T21:59:34.487575+0000","last_became_active":"2024-03-11T21:59:07.075187+0000","last_became_peered":"2024-03-11T21:59:07.075187+0000","last_unstale":"2024-03-11T21:59:34.487575+0000","last_undegraded":"2024-03-11T21:59:34.487575+0000","last_fullsized":"2024-03-11T21:59:34.487575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:45:23.775068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487906+0000","last_change":"2024-03-11T21:59:05.051540+0000","last_active":"2024-03-11T21:59:34.487906+0000","last_peered":"2024-03-11T21:59:34.487906+0000","last_clean":"2024-03-11T21:59:34.487906+0000","last_became_active":"2024-03-11T21:59:05.051224+0000","last_became_peered":"2024-03-11T21:59:05.051224+0000","last_unstale":"2024-03-11T21:59:34.487906+0000","last_undegraded":"2024-03-11T21:59:34.487906+0000","last_fullsized":"2024-03-11T21:59:34.487906+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:09:46.632361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489396+0000","last_change":"2024-03-11T21:59:01.040354+0000","last_active":"2024-03-11T21:59:34.489396+0000","last_peered":"2024-03-11T21:59:34.489396+0000","last_clean":"2024-03-11T21:59:34.489396+0000","last_became_active":"2024-03-11T21:59:01.040045+0000","last_became_peered":"2024-03-11T21:59:01.040045+0000","last_unstale":"2024-03-11T21:59:34.489396+0000","last_undegraded":"2024-03-11T21:59:34.489396+0000","last_fullsized":"2024-03-11T21:59:34.489396+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:36:13.634549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488109+0000","last_change":"2024-03-11T21:59:03.041716+0000","last_active":"2024-03-11T21:59:34.488109+0000","last_peered":"2024-03-11T21:59:34.488109+0000","last_clean":"2024-03-11T21:59:34.488109+0000","last_became_active":"2024-03-11T21:59:03.041389+0000","last_became_peered":"2024-03-11T21:59:03.041389+0000","last_unstale":"2024-03-11T21:59:34.488109+0000","last_undegraded":"2024-03-11T21:59:34.488109+0000","last_fullsized":"2024-03-11T21:59:34.488109+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:44:53.468648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486479+0000","last_change":"2024-03-11T21:59:07.079058+0000","last_active":"2024-03-11T21:59:34.486479+0000","last_peered":"2024-03-11T21:59:34.486479+0000","last_clean":"2024-03-11T21:59:34.486479+0000","last_became_active":"2024-03-11T21:59:07.078425+0000","last_became_peered":"2024-03-11T21:59:07.078425+0000","last_unstale":"2024-03-11T21:59:34.486479+0000","last_undegraded":"2024-03-11T21:59:34.486479+0000","last_fullsized":"2024-03-11T21:59:34.486479+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:22:50.747085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487485+0000","last_change":"2024-03-11T21:59:05.054646+0000","last_active":"2024-03-11T21:59:34.487485+0000","last_peered":"2024-03-11T21:59:34.487485+0000","last_clean":"2024-03-11T21:59:34.487485+0000","last_became_active":"2024-03-11T21:59:05.054042+0000","last_became_peered":"2024-03-11T21:59:05.054042+0000","last_unstale":"2024-03-11T21:59:34.487485+0000","last_undegraded":"2024-03-11T21:59:34.487485+0000","last_fullsized":"2024-03-11T21:59:34.487485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:53:27.471821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488779+0000","last_change":"2024-03-11T21:59:01.039314+0000","last_active":"2024-03-11T21:59:34.488779+0000","last_peered":"2024-03-11T21:59:34.488779+0000","last_clean":"2024-03-11T21:59:34.488779+0000","last_became_active":"2024-03-11T21:59:01.039031+0000","last_became_peered":"2024-03-11T21:59:01.039031+0000","last_unstale":"2024-03-11T21:59:34.488779+0000","last_undegraded":"2024-03-11T21:59:34.488779+0000","last_fullsized":"2024-03-11T21:59:34.488779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:56.584487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489116+0000","last_change":"2024-03-11T21:59:03.044410+0000","last_active":"2024-03-11T21:59:34.489116+0000","last_peered":"2024-03-11T21:59:34.489116+0000","last_clean":"2024-03-11T21:59:34.489116+0000","last_became_active":"2024-03-11T21:59:03.044107+0000","last_became_peered":"2024-03-11T21:59:03.044107+0000","last_unstale":"2024-03-11T21:59:34.489116+0000","last_undegraded":"2024-03-11T21:59:34.489116+0000","last_fullsized":"2024-03-11T21:59:34.489116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:51.957904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488129+0000","last_change":"2024-03-11T21:59:07.069069+0000","last_active":"2024-03-11T21:59:34.488129+0000","last_peered":"2024-03-11T21:59:34.488129+0000","last_clean":"2024-03-11T21:59:34.488129+0000","last_became_active":"2024-03-11T21:59:07.068774+0000","last_became_peered":"2024-03-11T21:59:07.068774+0000","last_unstale":"2024-03-11T21:59:34.488129+0000","last_undegraded":"2024-03-11T21:59:34.488129+0000","last_fullsized":"2024-03-11T21:59:34.488129+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:00:48.022324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489147+0000","last_change":"2024-03-11T21:59:05.054709+0000","last_active":"2024-03-11T21:59:34.489147+0000","last_peered":"2024-03-11T21:59:34.489147+0000","last_clean":"2024-03-11T21:59:34.489147+0000","last_became_active":"2024-03-11T21:59:05.053399+0000","last_became_peered":"2024-03-11T21:59:05.053399+0000","last_unstale":"2024-03-11T21:59:34.489147+0000","last_undegraded":"2024-03-11T21:59:34.489147+0000","last_fullsized":"2024-03-11T21:59:34.489147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:04.326875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489836+0000","last_change":"2024-03-11T21:59:01.039388+0000","last_active":"2024-03-11T21:59:34.489836+0000","last_peered":"2024-03-11T21:59:34.489836+0000","last_clean":"2024-03-11T21:59:34.489836+0000","last_became_active":"2024-03-11T21:59:01.038640+0000","last_became_peered":"2024-03-11T21:59:01.038640+0000","last_unstale":"2024-03-11T21:59:34.489836+0000","last_undegraded":"2024-03-11T21:59:34.489836+0000","last_fullsized":"2024-03-11T21:59:34.489836+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:35:05.934298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488805+0000","last_change":"2024-03-11T21:59:03.041115+0000","last_active":"2024-03-11T21:59:34.488805+0000","last_peered":"2024-03-11T21:59:34.488805+0000","last_clean":"2024-03-11T21:59:34.488805+0000","last_became_active":"2024-03-11T21:59:03.040615+0000","last_became_peered":"2024-03-11T21:59:03.040615+0000","last_unstale":"2024-03-11T21:59:34.488805+0000","last_undegraded":"2024-03-11T21:59:34.488805+0000","last_fullsized":"2024-03-11T21:59:34.488805+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:29.602749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488330+0000","last_change":"2024-03-11T21:59:07.067463+0000","last_active":"2024-03-11T21:59:34.488330+0000","last_peered":"2024-03-11T21:59:34.488330+0000","last_clean":"2024-03-11T21:59:34.488330+0000","last_became_active":"2024-03-11T21:59:07.067122+0000","last_became_peered":"2024-03-11T21:59:07.067122+0000","last_unstale":"2024-03-11T21:59:34.488330+0000","last_undegraded":"2024-03-11T21:59:34.488330+0000","last_fullsized":"2024-03-11T21:59:34.488330+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:04.423427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488007+0000","last_change":"2024-03-11T21:59:05.054233+0000","last_active":"2024-03-11T21:59:34.488007+0000","last_peered":"2024-03-11T21:59:34.488007+0000","last_clean":"2024-03-11T21:59:34.488007+0000","last_became_active":"2024-03-11T21:59:05.052959+0000","last_became_peered":"2024-03-11T21:59:05.052959+0000","last_unstale":"2024-03-11T21:59:34.488007+0000","last_undegraded":"2024-03-11T21:59:34.488007+0000","last_fullsized":"2024-03-11T21:59:34.488007+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:24.691679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490831+0000","last_change":"2024-03-11T21:59:01.038993+0000","last_active":"2024-03-11T21:59:34.490831+0000","last_peered":"2024-03-11T21:59:34.490831+0000","last_clean":"2024-03-11T21:59:34.490831+0000","last_became_active":"2024-03-11T21:59:01.038611+0000","last_became_peered":"2024-03-11T21:59:01.038611+0000","last_unstale":"2024-03-11T21:59:34.490831+0000","last_undegraded":"2024-03-11T21:59:34.490831+0000","last_fullsized":"2024-03-11T21:59:34.490831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:38:01.953125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489621+0000","last_change":"2024-03-11T21:59:03.047975+0000","last_active":"2024-03-11T21:59:34.489621+0000","last_peered":"2024-03-11T21:59:34.489621+0000","last_clean":"2024-03-11T21:59:34.489621+0000","last_became_active":"2024-03-11T21:59:03.047684+0000","last_became_peered":"2024-03-11T21:59:03.047684+0000","last_unstale":"2024-03-11T21:59:34.489621+0000","last_undegraded":"2024-03-11T21:59:34.489621+0000","last_fullsized":"2024-03-11T21:59:34.489621+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:20.969101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488441+0000","last_change":"2024-03-11T21:59:07.069304+0000","last_active":"2024-03-11T21:59:34.488441+0000","last_peered":"2024-03-11T21:59:34.488441+0000","last_clean":"2024-03-11T21:59:34.488441+0000","last_became_active":"2024-03-11T21:59:07.068544+0000","last_became_peered":"2024-03-11T21:59:07.068544+0000","last_unstale":"2024-03-11T21:59:34.488441+0000","last_undegraded":"2024-03-11T21:59:34.488441+0000","last_fullsized":"2024-03-11T21:59:34.488441+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:47.017277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489011+0000","last_change":"2024-03-11T21:59:05.056872+0000","last_active":"2024-03-11T21:59:34.489011+0000","last_peered":"2024-03-11T21:59:34.489011+0000","last_clean":"2024-03-11T21:59:34.489011+0000","last_became_active":"2024-03-11T21:59:05.056591+0000","last_became_peered":"2024-03-11T21:59:05.056591+0000","last_unstale":"2024-03-11T21:59:34.489011+0000","last_undegraded":"2024-03-11T21:59:34.489011+0000","last_fullsized":"2024-03-11T21:59:34.489011+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:10.415552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490598+0000","last_change":"2024-03-11T21:59:01.036174+0000","last_active":"2024-03-11T21:59:34.490598+0000","last_peered":"2024-03-11T21:59:34.490598+0000","last_clean":"2024-03-11T21:59:34.490598+0000","last_became_active":"2024-03-11T21:59:01.035879+0000","last_became_peered":"2024-03-11T21:59:01.035879+0000","last_unstale":"2024-03-11T21:59:34.490598+0000","last_undegraded":"2024-03-11T21:59:34.490598+0000","last_fullsized":"2024-03-11T21:59:34.490598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:34.211387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489059+0000","last_change":"2024-03-11T21:59:03.046831+0000","last_active":"2024-03-11T21:59:34.489059+0000","last_peered":"2024-03-11T21:59:34.489059+0000","last_clean":"2024-03-11T21:59:34.489059+0000","last_became_active":"2024-03-11T21:59:03.046539+0000","last_became_peered":"2024-03-11T21:59:03.046539+0000","last_unstale":"2024-03-11T21:59:34.489059+0000","last_undegraded":"2024-03-11T21:59:34.489059+0000","last_fullsized":"2024-03-11T21:59:34.489059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:54.629296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487865+0000","last_change":"2024-03-11T21:59:07.077512+0000","last_active":"2024-03-11T21:59:34.487865+0000","last_peered":"2024-03-11T21:59:34.487865+0000","last_clean":"2024-03-11T21:59:34.487865+0000","last_became_active":"2024-03-11T21:59:07.077118+0000","last_became_peered":"2024-03-11T21:59:07.077118+0000","last_unstale":"2024-03-11T21:59:34.487865+0000","last_undegraded":"2024-03-11T21:59:34.487865+0000","last_fullsized":"2024-03-11T21:59:34.487865+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:22.796422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489421+0000","last_change":"2024-03-11T21:59:05.056343+0000","last_active":"2024-03-11T21:59:34.489421+0000","last_peered":"2024-03-11T21:59:34.489421+0000","last_clean":"2024-03-11T21:59:34.489421+0000","last_became_active":"2024-03-11T21:59:05.056003+0000","last_became_peered":"2024-03-11T21:59:05.056003+0000","last_unstale":"2024-03-11T21:59:34.489421+0000","last_undegraded":"2024-03-11T21:59:34.489421+0000","last_fullsized":"2024-03-11T21:59:34.489421+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:06.300941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488318+0000","last_change":"2024-03-11T21:59:01.037389+0000","last_active":"2024-03-11T21:59:34.488318+0000","last_peered":"2024-03-11T21:59:34.488318+0000","last_clean":"2024-03-11T21:59:34.488318+0000","last_became_active":"2024-03-11T21:59:01.036598+0000","last_became_peered":"2024-03-11T21:59:01.036598+0000","last_unstale":"2024-03-11T21:59:34.488318+0000","last_undegraded":"2024-03-11T21:59:34.488318+0000","last_fullsized":"2024-03-11T21:59:34.488318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:02.005804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488847+0000","last_change":"2024-03-11T21:59:03.047801+0000","last_active":"2024-03-11T21:59:34.488847+0000","last_peered":"2024-03-11T21:59:34.488847+0000","last_clean":"2024-03-11T21:59:34.488847+0000","last_became_active":"2024-03-11T21:59:03.047493+0000","last_became_peered":"2024-03-11T21:59:03.047493+0000","last_unstale":"2024-03-11T21:59:34.488847+0000","last_undegraded":"2024-03-11T21:59:34.488847+0000","last_fullsized":"2024-03-11T21:59:34.488847+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:25.216397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487528+0000","last_change":"2024-03-11T21:59:07.077573+0000","last_active":"2024-03-11T21:59:34.487528+0000","last_peered":"2024-03-11T21:59:34.487528+0000","last_clean":"2024-03-11T21:59:34.487528+0000","last_became_active":"2024-03-11T21:59:07.077213+0000","last_became_peered":"2024-03-11T21:59:07.077213+0000","last_unstale":"2024-03-11T21:59:34.487528+0000","last_undegraded":"2024-03-11T21:59:34.487528+0000","last_fullsized":"2024-03-11T21:59:34.487528+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:47:00.146940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087862+0000","last_change":"2024-03-11T21:59:05.056491+0000","last_active":"2024-03-11T21:59:36.087862+0000","last_peered":"2024-03-11T21:59:36.087862+0000","last_clean":"2024-03-11T21:59:36.087862+0000","last_became_active":"2024-03-11T21:59:05.056186+0000","last_became_peered":"2024-03-11T21:59:05.056186+0000","last_unstale":"2024-03-11T21:59:36.087862+0000","last_undegraded":"2024-03-11T21:59:36.087862+0000","last_fullsized":"2024-03-11T21:59:36.087862+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:07:59.936686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486777+0000","last_change":"2024-03-11T21:59:01.039950+0000","last_active":"2024-03-11T21:59:34.486777+0000","last_peered":"2024-03-11T21:59:34.486777+0000","last_clean":"2024-03-11T21:59:34.486777+0000","last_became_active":"2024-03-11T21:59:01.039041+0000","last_became_peered":"2024-03-11T21:59:01.039041+0000","last_unstale":"2024-03-11T21:59:34.486777+0000","last_undegraded":"2024-03-11T21:59:34.486777+0000","last_fullsized":"2024-03-11T21:59:34.486777+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:44:12.294260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486204+0000","last_change":"2024-03-11T21:59:03.041612+0000","last_active":"2024-03-11T21:59:34.486204+0000","last_peered":"2024-03-11T21:59:34.486204+0000","last_clean":"2024-03-11T21:59:34.486204+0000","last_became_active":"2024-03-11T21:59:03.041309+0000","last_became_peered":"2024-03-11T21:59:03.041309+0000","last_unstale":"2024-03-11T21:59:34.486204+0000","last_undegraded":"2024-03-11T21:59:34.486204+0000","last_fullsized":"2024-03-11T21:59:34.486204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:52:18.163209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488987+0000","last_change":"2024-03-11T21:59:07.076528+0000","last_active":"2024-03-11T21:59:34.488987+0000","last_peered":"2024-03-11T21:59:34.488987+0000","last_clean":"2024-03-11T21:59:34.488987+0000","last_became_active":"2024-03-11T21:59:07.076163+0000","last_became_peered":"2024-03-11T21:59:07.076163+0000","last_unstale":"2024-03-11T21:59:34.488987+0000","last_undegraded":"2024-03-11T21:59:34.488987+0000","last_fullsized":"2024-03-11T21:59:34.488987+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:05:58.226928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487707+0000","last_change":"2024-03-11T21:59:05.054816+0000","last_active":"2024-03-11T21:59:34.487707+0000","last_peered":"2024-03-11T21:59:34.487707+0000","last_clean":"2024-03-11T21:59:34.487707+0000","last_became_active":"2024-03-11T21:59:05.053588+0000","last_became_peered":"2024-03-11T21:59:05.053588+0000","last_unstale":"2024-03-11T21:59:34.487707+0000","last_undegraded":"2024-03-11T21:59:34.487707+0000","last_fullsized":"2024-03-11T21:59:34.487707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:23:16.723768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488067+0000","last_change":"2024-03-11T21:59:01.040373+0000","last_active":"2024-03-11T21:59:34.488067+0000","last_peered":"2024-03-11T21:59:34.488067+0000","last_clean":"2024-03-11T21:59:34.488067+0000","last_became_active":"2024-03-11T21:59:01.040056+0000","last_became_peered":"2024-03-11T21:59:01.040056+0000","last_unstale":"2024-03-11T21:59:34.488067+0000","last_undegraded":"2024-03-11T21:59:34.488067+0000","last_fullsized":"2024-03-11T21:59:34.488067+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:03:47.842368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486455+0000","last_change":"2024-03-11T21:59:03.041541+0000","last_active":"2024-03-11T21:59:34.486455+0000","last_peered":"2024-03-11T21:59:34.486455+0000","last_clean":"2024-03-11T21:59:34.486455+0000","last_became_active":"2024-03-11T21:59:03.041266+0000","last_became_peered":"2024-03-11T21:59:03.041266+0000","last_unstale":"2024-03-11T21:59:34.486455+0000","last_undegraded":"2024-03-11T21:59:34.486455+0000","last_fullsized":"2024-03-11T21:59:34.486455+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.232039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487647+0000","last_change":"2024-03-11T21:59:07.076660+0000","last_active":"2024-03-11T21:59:34.487647+0000","last_peered":"2024-03-11T21:59:34.487647+0000","last_clean":"2024-03-11T21:59:34.487647+0000","last_became_active":"2024-03-11T21:59:07.076328+0000","last_became_peered":"2024-03-11T21:59:07.076328+0000","last_unstale":"2024-03-11T21:59:34.487647+0000","last_undegraded":"2024-03-11T21:59:34.487647+0000","last_fullsized":"2024-03-11T21:59:34.487647+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:30.305216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486375+0000","last_change":"2024-03-11T21:59:05.054252+0000","last_active":"2024-03-11T21:59:34.486375+0000","last_peered":"2024-03-11T21:59:34.486375+0000","last_clean":"2024-03-11T21:59:34.486375+0000","last_became_active":"2024-03-11T21:59:05.053418+0000","last_became_peered":"2024-03-11T21:59:05.053418+0000","last_unstale":"2024-03-11T21:59:34.486375+0000","last_undegraded":"2024-03-11T21:59:34.486375+0000","last_fullsized":"2024-03-11T21:59:34.486375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:00:26.879610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489767+0000","last_change":"2024-03-11T21:59:01.040450+0000","last_active":"2024-03-11T21:59:34.489767+0000","last_peered":"2024-03-11T21:59:34.489767+0000","last_clean":"2024-03-11T21:59:34.489767+0000","last_became_active":"2024-03-11T21:59:01.039996+0000","last_became_peered":"2024-03-11T21:59:01.039996+0000","last_unstale":"2024-03-11T21:59:34.489767+0000","last_undegraded":"2024-03-11T21:59:34.489767+0000","last_fullsized":"2024-03-11T21:59:34.489767+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:10.643347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489965+0000","last_change":"2024-03-11T21:59:03.042457+0000","last_active":"2024-03-11T21:59:34.489965+0000","last_peered":"2024-03-11T21:59:34.489965+0000","last_clean":"2024-03-11T21:59:34.489965+0000","last_became_active":"2024-03-11T21:59:03.042105+0000","last_became_peered":"2024-03-11T21:59:03.042105+0000","last_unstale":"2024-03-11T21:59:34.489965+0000","last_undegraded":"2024-03-11T21:59:34.489965+0000","last_fullsized":"2024-03-11T21:59:34.489965+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:37.463970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487539+0000","last_change":"2024-03-11T21:59:07.067478+0000","last_active":"2024-03-11T21:59:34.487539+0000","last_peered":"2024-03-11T21:59:34.487539+0000","last_clean":"2024-03-11T21:59:34.487539+0000","last_became_active":"2024-03-11T21:59:07.067076+0000","last_became_peered":"2024-03-11T21:59:07.067076+0000","last_unstale":"2024-03-11T21:59:34.487539+0000","last_undegraded":"2024-03-11T21:59:34.487539+0000","last_fullsized":"2024-03-11T21:59:34.487539+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:04.533060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486522+0000","last_change":"2024-03-11T21:59:05.052745+0000","last_active":"2024-03-11T21:59:34.486522+0000","last_peered":"2024-03-11T21:59:34.486522+0000","last_clean":"2024-03-11T21:59:34.486522+0000","last_became_active":"2024-03-11T21:59:05.051792+0000","last_became_peered":"2024-03-11T21:59:05.051792+0000","last_unstale":"2024-03-11T21:59:34.486522+0000","last_undegraded":"2024-03-11T21:59:34.486522+0000","last_fullsized":"2024-03-11T21:59:34.486522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:00:08.381923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489647+0000","last_change":"2024-03-11T21:59:01.037649+0000","last_active":"2024-03-11T21:59:34.489647+0000","last_peered":"2024-03-11T21:59:34.489647+0000","last_clean":"2024-03-11T21:59:34.489647+0000","last_became_active":"2024-03-11T21:59:01.036686+0000","last_became_peered":"2024-03-11T21:59:01.036686+0000","last_unstale":"2024-03-11T21:59:34.489647+0000","last_undegraded":"2024-03-11T21:59:34.489647+0000","last_fullsized":"2024-03-11T21:59:34.489647+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:53.644624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489254+0000","last_change":"2024-03-11T21:59:03.044372+0000","last_active":"2024-03-11T21:59:34.489254+0000","last_peered":"2024-03-11T21:59:34.489254+0000","last_clean":"2024-03-11T21:59:34.489254+0000","last_became_active":"2024-03-11T21:59:03.044064+0000","last_became_peered":"2024-03-11T21:59:03.044064+0000","last_unstale":"2024-03-11T21:59:34.489254+0000","last_undegraded":"2024-03-11T21:59:34.489254+0000","last_fullsized":"2024-03-11T21:59:34.489254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:26:36.666112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488356+0000","last_change":"2024-03-11T21:59:07.069658+0000","last_active":"2024-03-11T21:59:34.488356+0000","last_peered":"2024-03-11T21:59:34.488356+0000","last_clean":"2024-03-11T21:59:34.488356+0000","last_became_active":"2024-03-11T21:59:07.069285+0000","last_became_peered":"2024-03-11T21:59:07.069285+0000","last_unstale":"2024-03-11T21:59:34.488356+0000","last_undegraded":"2024-03-11T21:59:34.488356+0000","last_fullsized":"2024-03-11T21:59:34.488356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:09.893670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488674+0000","last_change":"2024-03-11T21:59:05.054875+0000","last_active":"2024-03-11T21:59:34.488674+0000","last_peered":"2024-03-11T21:59:34.488674+0000","last_clean":"2024-03-11T21:59:34.488674+0000","last_became_active":"2024-03-11T21:59:05.054487+0000","last_became_peered":"2024-03-11T21:59:05.054487+0000","last_unstale":"2024-03-11T21:59:34.488674+0000","last_undegraded":"2024-03-11T21:59:34.488674+0000","last_fullsized":"2024-03-11T21:59:34.488674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:59.856730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489529+0000","last_change":"2024-03-11T21:59:01.039665+0000","last_active":"2024-03-11T21:59:34.489529+0000","last_peered":"2024-03-11T21:59:34.489529+0000","last_clean":"2024-03-11T21:59:34.489529+0000","last_became_active":"2024-03-11T21:59:01.039027+0000","last_became_peered":"2024-03-11T21:59:01.039027+0000","last_unstale":"2024-03-11T21:59:34.489529+0000","last_undegraded":"2024-03-11T21:59:34.489529+0000","last_fullsized":"2024-03-11T21:59:34.489529+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:32.093622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489023+0000","last_change":"2024-03-11T21:59:03.042669+0000","last_active":"2024-03-11T21:59:34.489023+0000","last_peered":"2024-03-11T21:59:34.489023+0000","last_clean":"2024-03-11T21:59:34.489023+0000","last_became_active":"2024-03-11T21:59:03.041986+0000","last_became_peered":"2024-03-11T21:59:03.041986+0000","last_unstale":"2024-03-11T21:59:34.489023+0000","last_undegraded":"2024-03-11T21:59:34.489023+0000","last_fullsized":"2024-03-11T21:59:34.489023+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:10.531645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488176+0000","last_change":"2024-03-11T21:59:07.069211+0000","last_active":"2024-03-11T21:59:34.488176+0000","last_peered":"2024-03-11T21:59:34.488176+0000","last_clean":"2024-03-11T21:59:34.488176+0000","last_became_active":"2024-03-11T21:59:07.068840+0000","last_became_peered":"2024-03-11T21:59:07.068840+0000","last_unstale":"2024-03-11T21:59:34.488176+0000","last_undegraded":"2024-03-11T21:59:34.488176+0000","last_fullsized":"2024-03-11T21:59:34.488176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:57.300601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486660+0000","last_change":"2024-03-11T21:59:05.054401+0000","last_active":"2024-03-11T21:59:34.486660+0000","last_peered":"2024-03-11T21:59:34.486660+0000","last_clean":"2024-03-11T21:59:34.486660+0000","last_became_active":"2024-03-11T21:59:05.053501+0000","last_became_peered":"2024-03-11T21:59:05.053501+0000","last_unstale":"2024-03-11T21:59:34.486660+0000","last_undegraded":"2024-03-11T21:59:34.486660+0000","last_fullsized":"2024-03-11T21:59:34.486660+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:00:56.193895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488676+0000","last_change":"2024-03-11T21:59:01.036618+0000","last_active":"2024-03-11T21:59:34.488676+0000","last_peered":"2024-03-11T21:59:34.488676+0000","last_clean":"2024-03-11T21:59:34.488676+0000","last_became_active":"2024-03-11T21:59:01.036043+0000","last_became_peered":"2024-03-11T21:59:01.036043+0000","last_unstale":"2024-03-11T21:59:34.488676+0000","last_undegraded":"2024-03-11T21:59:34.488676+0000","last_fullsized":"2024-03-11T21:59:34.488676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:29:33.184690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489295+0000","last_change":"2024-03-11T21:59:03.046920+0000","last_active":"2024-03-11T21:59:34.489295+0000","last_peered":"2024-03-11T21:59:34.489295+0000","last_clean":"2024-03-11T21:59:34.489295+0000","last_became_active":"2024-03-11T21:59:03.046607+0000","last_became_peered":"2024-03-11T21:59:03.046607+0000","last_unstale":"2024-03-11T21:59:34.489295+0000","last_undegraded":"2024-03-11T21:59:34.489295+0000","last_fullsized":"2024-03-11T21:59:34.489295+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:37:18.267340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488221+0000","last_change":"2024-03-11T21:59:07.078856+0000","last_active":"2024-03-11T21:59:34.488221+0000","last_peered":"2024-03-11T21:59:34.488221+0000","last_clean":"2024-03-11T21:59:34.488221+0000","last_became_active":"2024-03-11T21:59:07.078264+0000","last_became_peered":"2024-03-11T21:59:07.078264+0000","last_unstale":"2024-03-11T21:59:34.488221+0000","last_undegraded":"2024-03-11T21:59:34.488221+0000","last_fullsized":"2024-03-11T21:59:34.488221+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:54:12.656535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087828+0000","last_change":"2024-03-11T21:59:05.056760+0000","last_active":"2024-03-11T21:59:36.087828+0000","last_peered":"2024-03-11T21:59:36.087828+0000","last_clean":"2024-03-11T21:59:36.087828+0000","last_became_active":"2024-03-11T21:59:05.056179+0000","last_became_peered":"2024-03-11T21:59:05.056179+0000","last_unstale":"2024-03-11T21:59:36.087828+0000","last_undegraded":"2024-03-11T21:59:36.087828+0000","last_fullsized":"2024-03-11T21:59:36.087828+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:29:17.460397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488616+0000","last_change":"2024-03-11T21:59:01.036133+0000","last_active":"2024-03-11T21:59:34.488616+0000","last_peered":"2024-03-11T21:59:34.488616+0000","last_clean":"2024-03-11T21:59:34.488616+0000","last_became_active":"2024-03-11T21:59:01.035644+0000","last_became_peered":"2024-03-11T21:59:01.035644+0000","last_unstale":"2024-03-11T21:59:34.488616+0000","last_undegraded":"2024-03-11T21:59:34.488616+0000","last_fullsized":"2024-03-11T21:59:34.488616+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:16:38.487046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486983+0000","last_change":"2024-03-11T21:59:03.040782+0000","last_active":"2024-03-11T21:59:34.486983+0000","last_peered":"2024-03-11T21:59:34.486983+0000","last_clean":"2024-03-11T21:59:34.486983+0000","last_became_active":"2024-03-11T21:59:03.040534+0000","last_became_peered":"2024-03-11T21:59:03.040534+0000","last_unstale":"2024-03-11T21:59:34.486983+0000","last_undegraded":"2024-03-11T21:59:34.486983+0000","last_fullsized":"2024-03-11T21:59:34.486983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:10:32.311227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488989+0000","last_change":"2024-03-11T21:59:07.067674+0000","last_active":"2024-03-11T21:59:34.488989+0000","last_peered":"2024-03-11T21:59:34.488989+0000","last_clean":"2024-03-11T21:59:34.488989+0000","last_became_active":"2024-03-11T21:59:07.067275+0000","last_became_peered":"2024-03-11T21:59:07.067275+0000","last_unstale":"2024-03-11T21:59:34.488989+0000","last_undegraded":"2024-03-11T21:59:34.488989+0000","last_fullsized":"2024-03-11T21:59:34.488989+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:51.192763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087845+0000","last_change":"2024-03-11T21:59:05.055282+0000","last_active":"2024-03-11T21:59:36.087845+0000","last_peered":"2024-03-11T21:59:36.087845+0000","last_clean":"2024-03-11T21:59:36.087845+0000","last_became_active":"2024-03-11T21:59:05.054882+0000","last_became_peered":"2024-03-11T21:59:05.054882+0000","last_unstale":"2024-03-11T21:59:36.087845+0000","last_undegraded":"2024-03-11T21:59:36.087845+0000","last_fullsized":"2024-03-11T21:59:36.087845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:47:30.146828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488604+0000","last_change":"2024-03-11T21:59:01.038760+0000","last_active":"2024-03-11T21:59:34.488604+0000","last_peered":"2024-03-11T21:59:34.488604+0000","last_clean":"2024-03-11T21:59:34.488604+0000","last_became_active":"2024-03-11T21:59:01.038373+0000","last_became_peered":"2024-03-11T21:59:01.038373+0000","last_unstale":"2024-03-11T21:59:34.488604+0000","last_undegraded":"2024-03-11T21:59:34.488604+0000","last_fullsized":"2024-03-11T21:59:34.488604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:01:25.523518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488995+0000","last_change":"2024-03-11T21:59:03.045811+0000","last_active":"2024-03-11T21:59:34.488995+0000","last_peered":"2024-03-11T21:59:34.488995+0000","last_clean":"2024-03-11T21:59:34.488995+0000","last_became_active":"2024-03-11T21:59:03.045390+0000","last_became_peered":"2024-03-11T21:59:03.045390+0000","last_unstale":"2024-03-11T21:59:34.488995+0000","last_undegraded":"2024-03-11T21:59:34.488995+0000","last_fullsized":"2024-03-11T21:59:34.488995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:00.728036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489184+0000","last_change":"2024-03-11T21:59:07.069040+0000","last_active":"2024-03-11T21:59:34.489184+0000","last_peered":"2024-03-11T21:59:34.489184+0000","last_clean":"2024-03-11T21:59:34.489184+0000","last_became_active":"2024-03-11T21:59:07.068717+0000","last_became_peered":"2024-03-11T21:59:07.068717+0000","last_unstale":"2024-03-11T21:59:34.489184+0000","last_undegraded":"2024-03-11T21:59:34.489184+0000","last_fullsized":"2024-03-11T21:59:34.489184+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:48:25.192503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489030+0000","last_change":"2024-03-11T21:59:05.054655+0000","last_active":"2024-03-11T21:59:34.489030+0000","last_peered":"2024-03-11T21:59:34.489030+0000","last_clean":"2024-03-11T21:59:34.489030+0000","last_became_active":"2024-03-11T21:59:05.053654+0000","last_became_peered":"2024-03-11T21:59:05.053654+0000","last_unstale":"2024-03-11T21:59:34.489030+0000","last_undegraded":"2024-03-11T21:59:34.489030+0000","last_fullsized":"2024-03-11T21:59:34.489030+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:16:32.862281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487652+0000","last_change":"2024-03-11T21:59:01.040250+0000","last_active":"2024-03-11T21:59:34.487652+0000","last_peered":"2024-03-11T21:59:34.487652+0000","last_clean":"2024-03-11T21:59:34.487652+0000","last_became_active":"2024-03-11T21:59:01.039866+0000","last_became_peered":"2024-03-11T21:59:01.039866+0000","last_unstale":"2024-03-11T21:59:34.487652+0000","last_undegraded":"2024-03-11T21:59:34.487652+0000","last_fullsized":"2024-03-11T21:59:34.487652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:29.952287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487223+0000","last_change":"2024-03-11T21:59:03.041103+0000","last_active":"2024-03-11T21:59:34.487223+0000","last_peered":"2024-03-11T21:59:34.487223+0000","last_clean":"2024-03-11T21:59:34.487223+0000","last_became_active":"2024-03-11T21:59:03.040742+0000","last_became_peered":"2024-03-11T21:59:03.040742+0000","last_unstale":"2024-03-11T21:59:34.487223+0000","last_undegraded":"2024-03-11T21:59:34.487223+0000","last_fullsized":"2024-03-11T21:59:34.487223+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:51:34.613463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487551+0000","last_change":"2024-03-11T21:59:07.077576+0000","last_active":"2024-03-11T21:59:34.487551+0000","last_peered":"2024-03-11T21:59:34.487551+0000","last_clean":"2024-03-11T21:59:34.487551+0000","last_became_active":"2024-03-11T21:59:07.077208+0000","last_became_peered":"2024-03-11T21:59:07.077208+0000","last_unstale":"2024-03-11T21:59:34.487551+0000","last_undegraded":"2024-03-11T21:59:34.487551+0000","last_fullsized":"2024-03-11T21:59:34.487551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:32.921859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487797+0000","last_change":"2024-03-11T21:59:05.055377+0000","last_active":"2024-03-11T21:59:34.487797+0000","last_peered":"2024-03-11T21:59:34.487797+0000","last_clean":"2024-03-11T21:59:34.487797+0000","last_became_active":"2024-03-11T21:59:05.054747+0000","last_became_peered":"2024-03-11T21:59:05.054747+0000","last_unstale":"2024-03-11T21:59:34.487797+0000","last_undegraded":"2024-03-11T21:59:34.487797+0000","last_fullsized":"2024-03-11T21:59:34.487797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:30:08.968548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488243+0000","last_change":"2024-03-11T21:59:01.040374+0000","last_active":"2024-03-11T21:59:34.488243+0000","last_peered":"2024-03-11T21:59:34.488243+0000","last_clean":"2024-03-11T21:59:34.488243+0000","last_became_active":"2024-03-11T21:59:01.040042+0000","last_became_peered":"2024-03-11T21:59:01.040042+0000","last_unstale":"2024-03-11T21:59:34.488243+0000","last_undegraded":"2024-03-11T21:59:34.488243+0000","last_fullsized":"2024-03-11T21:59:34.488243+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:53.091994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490962+0000","last_change":"2024-03-11T21:59:03.046665+0000","last_active":"2024-03-11T21:59:34.490962+0000","last_peered":"2024-03-11T21:59:34.490962+0000","last_clean":"2024-03-11T21:59:34.490962+0000","last_became_active":"2024-03-11T21:59:03.046390+0000","last_became_peered":"2024-03-11T21:59:03.046390+0000","last_unstale":"2024-03-11T21:59:34.490962+0000","last_undegraded":"2024-03-11T21:59:34.490962+0000","last_fullsized":"2024-03-11T21:59:34.490962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:44:32.521080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489932+0000","last_change":"2024-03-11T21:59:07.067571+0000","last_active":"2024-03-11T21:59:34.489932+0000","last_peered":"2024-03-11T21:59:34.489932+0000","last_clean":"2024-03-11T21:59:34.489932+0000","last_became_active":"2024-03-11T21:59:07.067352+0000","last_became_peered":"2024-03-11T21:59:07.067352+0000","last_unstale":"2024-03-11T21:59:34.489932+0000","last_undegraded":"2024-03-11T21:59:34.489932+0000","last_fullsized":"2024-03-11T21:59:34.489932+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:08.429269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489479+0000","last_change":"2024-03-11T21:59:01.039099+0000","last_active":"2024-03-11T21:59:34.489479+0000","last_peered":"2024-03-11T21:59:34.489479+0000","last_clean":"2024-03-11T21:59:34.489479+0000","last_became_active":"2024-03-11T21:59:01.038710+0000","last_became_peered":"2024-03-11T21:59:01.038710+0000","last_unstale":"2024-03-11T21:59:34.489479+0000","last_undegraded":"2024-03-11T21:59:34.489479+0000","last_fullsized":"2024-03-11T21:59:34.489479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:56.988219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490766+0000","last_change":"2024-03-11T21:59:03.048105+0000","last_active":"2024-03-11T21:59:34.490766+0000","last_peered":"2024-03-11T21:59:34.490766+0000","last_clean":"2024-03-11T21:59:34.490766+0000","last_became_active":"2024-03-11T21:59:03.047806+0000","last_became_peered":"2024-03-11T21:59:03.047806+0000","last_unstale":"2024-03-11T21:59:34.490766+0000","last_undegraded":"2024-03-11T21:59:34.490766+0000","last_fullsized":"2024-03-11T21:59:34.490766+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:45.989046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087825+0000","last_change":"2024-03-11T21:59:05.056835+0000","last_active":"2024-03-11T21:59:36.087825+0000","last_peered":"2024-03-11T21:59:36.087825+0000","last_clean":"2024-03-11T21:59:36.087825+0000","last_became_active":"2024-03-11T21:59:05.056414+0000","last_became_peered":"2024-03-11T21:59:05.056414+0000","last_unstale":"2024-03-11T21:59:36.087825+0000","last_undegraded":"2024-03-11T21:59:36.087825+0000","last_fullsized":"2024-03-11T21:59:36.087825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:10.862153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487211+0000","last_change":"2024-03-11T21:59:07.075162+0000","last_active":"2024-03-11T21:59:34.487211+0000","last_peered":"2024-03-11T21:59:34.487211+0000","last_clean":"2024-03-11T21:59:34.487211+0000","last_became_active":"2024-03-11T21:59:07.074789+0000","last_became_peered":"2024-03-11T21:59:07.074789+0000","last_unstale":"2024-03-11T21:59:34.487211+0000","last_undegraded":"2024-03-11T21:59:34.487211+0000","last_fullsized":"2024-03-11T21:59:34.487211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:46.090113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488662+0000","last_change":"2024-03-11T21:59:01.037870+0000","last_active":"2024-03-11T21:59:34.488662+0000","last_peered":"2024-03-11T21:59:34.488662+0000","last_clean":"2024-03-11T21:59:34.488662+0000","last_became_active":"2024-03-11T21:59:01.037513+0000","last_became_peered":"2024-03-11T21:59:01.037513+0000","last_unstale":"2024-03-11T21:59:34.488662+0000","last_undegraded":"2024-03-11T21:59:34.488662+0000","last_fullsized":"2024-03-11T21:59:34.488662+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:09:02.516378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087761+0000","last_change":"2024-03-11T21:59:03.042464+0000","last_active":"2024-03-11T21:59:36.087761+0000","last_peered":"2024-03-11T21:59:36.087761+0000","last_clean":"2024-03-11T21:59:36.087761+0000","last_became_active":"2024-03-11T21:59:03.041749+0000","last_became_peered":"2024-03-11T21:59:03.041749+0000","last_unstale":"2024-03-11T21:59:36.087761+0000","last_undegraded":"2024-03-11T21:59:36.087761+0000","last_fullsized":"2024-03-11T21:59:36.087761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:29:07.358472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488446+0000","last_change":"2024-03-11T21:59:05.055463+0000","last_active":"2024-03-11T21:59:34.488446+0000","last_peered":"2024-03-11T21:59:34.488446+0000","last_clean":"2024-03-11T21:59:34.488446+0000","last_became_active":"2024-03-11T21:59:05.054956+0000","last_became_peered":"2024-03-11T21:59:05.054956+0000","last_unstale":"2024-03-11T21:59:34.488446+0000","last_undegraded":"2024-03-11T21:59:34.488446+0000","last_fullsized":"2024-03-11T21:59:34.488446+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:18.678651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487322+0000","last_change":"2024-03-11T21:59:07.075208+0000","last_active":"2024-03-11T21:59:34.487322+0000","last_peered":"2024-03-11T21:59:34.487322+0000","last_clean":"2024-03-11T21:59:34.487322+0000","last_became_active":"2024-03-11T21:59:07.074858+0000","last_became_peered":"2024-03-11T21:59:07.074858+0000","last_unstale":"2024-03-11T21:59:34.487322+0000","last_undegraded":"2024-03-11T21:59:34.487322+0000","last_fullsized":"2024-03-11T21:59:34.487322+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:37.643897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488209+0000","last_change":"2024-03-11T21:59:01.038101+0000","last_active":"2024-03-11T21:59:34.488209+0000","last_peered":"2024-03-11T21:59:34.488209+0000","last_clean":"2024-03-11T21:59:34.488209+0000","last_became_active":"2024-03-11T21:59:01.037608+0000","last_became_peered":"2024-03-11T21:59:01.037608+0000","last_unstale":"2024-03-11T21:59:34.488209+0000","last_undegraded":"2024-03-11T21:59:34.488209+0000","last_fullsized":"2024-03-11T21:59:34.488209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:10.750688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489393+0000","last_change":"2024-03-11T21:59:03.046029+0000","last_active":"2024-03-11T21:59:34.489393+0000","last_peered":"2024-03-11T21:59:34.489393+0000","last_clean":"2024-03-11T21:59:34.489393+0000","last_became_active":"2024-03-11T21:59:03.045626+0000","last_became_peered":"2024-03-11T21:59:03.045626+0000","last_unstale":"2024-03-11T21:59:34.489393+0000","last_undegraded":"2024-03-11T21:59:34.489393+0000","last_fullsized":"2024-03-11T21:59:34.489393+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:07:49.827952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488379+0000","last_change":"2024-03-11T21:59:05.054927+0000","last_active":"2024-03-11T21:59:34.488379+0000","last_peered":"2024-03-11T21:59:34.488379+0000","last_clean":"2024-03-11T21:59:34.488379+0000","last_became_active":"2024-03-11T21:59:05.054387+0000","last_became_peered":"2024-03-11T21:59:05.054387+0000","last_unstale":"2024-03-11T21:59:34.488379+0000","last_undegraded":"2024-03-11T21:59:34.488379+0000","last_fullsized":"2024-03-11T21:59:34.488379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:15.472313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488896+0000","last_change":"2024-03-11T21:59:07.077996+0000","last_active":"2024-03-11T21:59:34.488896+0000","last_peered":"2024-03-11T21:59:34.488896+0000","last_clean":"2024-03-11T21:59:34.488896+0000","last_became_active":"2024-03-11T21:59:07.077608+0000","last_became_peered":"2024-03-11T21:59:07.077608+0000","last_unstale":"2024-03-11T21:59:34.488896+0000","last_undegraded":"2024-03-11T21:59:34.488896+0000","last_fullsized":"2024-03-11T21:59:34.488896+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:35.503888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487778+0000","last_change":"2024-03-11T21:59:03.047737+0000","last_active":"2024-03-11T21:59:34.487778+0000","last_peered":"2024-03-11T21:59:34.487778+0000","last_clean":"2024-03-11T21:59:34.487778+0000","last_became_active":"2024-03-11T21:59:03.047477+0000","last_became_peered":"2024-03-11T21:59:03.047477+0000","last_unstale":"2024-03-11T21:59:34.487778+0000","last_undegraded":"2024-03-11T21:59:34.487778+0000","last_fullsized":"2024-03-11T21:59:34.487778+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:40:52.913016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486667+0000","last_change":"2024-03-11T21:59:01.038820+0000","last_active":"2024-03-11T21:59:34.486667+0000","last_peered":"2024-03-11T21:59:34.486667+0000","last_clean":"2024-03-11T21:59:34.486667+0000","last_became_active":"2024-03-11T21:59:01.037953+0000","last_became_peered":"2024-03-11T21:59:01.037953+0000","last_unstale":"2024-03-11T21:59:34.486667+0000","last_undegraded":"2024-03-11T21:59:34.486667+0000","last_fullsized":"2024-03-11T21:59:34.486667+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:05:05.696632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489149+0000","last_change":"2024-03-11T21:59:05.056499+0000","last_active":"2024-03-11T21:59:34.489149+0000","last_peered":"2024-03-11T21:59:34.489149+0000","last_clean":"2024-03-11T21:59:34.489149+0000","last_became_active":"2024-03-11T21:59:05.055049+0000","last_became_peered":"2024-03-11T21:59:05.055049+0000","last_unstale":"2024-03-11T21:59:34.489149+0000","last_undegraded":"2024-03-11T21:59:34.489149+0000","last_fullsized":"2024-03-11T21:59:34.489149+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:46.434743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488441+0000","last_change":"2024-03-11T21:59:07.069421+0000","last_active":"2024-03-11T21:59:34.488441+0000","last_peered":"2024-03-11T21:59:34.488441+0000","last_clean":"2024-03-11T21:59:34.488441+0000","last_became_active":"2024-03-11T21:59:07.068925+0000","last_became_peered":"2024-03-11T21:59:07.068925+0000","last_unstale":"2024-03-11T21:59:34.488441+0000","last_undegraded":"2024-03-11T21:59:34.488441+0000","last_fullsized":"2024-03-11T21:59:34.488441+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:55:51.345506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486761+0000","last_change":"2024-03-11T21:59:03.042114+0000","last_active":"2024-03-11T21:59:34.486761+0000","last_peered":"2024-03-11T21:59:34.486761+0000","last_clean":"2024-03-11T21:59:34.486761+0000","last_became_active":"2024-03-11T21:59:03.041739+0000","last_became_peered":"2024-03-11T21:59:03.041739+0000","last_unstale":"2024-03-11T21:59:34.486761+0000","last_undegraded":"2024-03-11T21:59:34.486761+0000","last_fullsized":"2024-03-11T21:59:34.486761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:24.738017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489124+0000","last_change":"2024-03-11T21:59:01.041002+0000","last_active":"2024-03-11T21:59:34.489124+0000","last_peered":"2024-03-11T21:59:34.489124+0000","last_clean":"2024-03-11T21:59:34.489124+0000","last_became_active":"2024-03-11T21:59:01.040553+0000","last_became_peered":"2024-03-11T21:59:01.040553+0000","last_unstale":"2024-03-11T21:59:34.489124+0000","last_undegraded":"2024-03-11T21:59:34.489124+0000","last_fullsized":"2024-03-11T21:59:34.489124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:42:44.108320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.591324+0000","last_change":"2024-03-11T21:58:46.478599+0000","last_active":"2024-03-11T21:59:36.591324+0000","last_peered":"2024-03-11T21:59:36.591324+0000","last_clean":"2024-03-11T21:59:36.591324+0000","last_became_active":"2024-03-11T21:58:46.478232+0000","last_became_peered":"2024-03-11T21:58:46.478232+0000","last_unstale":"2024-03-11T21:59:36.591324+0000","last_undegraded":"2024-03-11T21:59:36.591324+0000","last_fullsized":"2024-03-11T21:59:36.591324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:58:45.465824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:58:45.465824+0000","last_clean_scrub_stamp":"2024-03-11T21:58:45.465824+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:45.199571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489745+0000","last_change":"2024-03-11T21:59:05.054324+0000","last_active":"2024-03-11T21:59:34.489745+0000","last_peered":"2024-03-11T21:59:34.489745+0000","last_clean":"2024-03-11T21:59:34.489745+0000","last_became_active":"2024-03-11T21:59:05.053974+0000","last_became_peered":"2024-03-11T21:59:05.053974+0000","last_unstale":"2024-03-11T21:59:34.489745+0000","last_undegraded":"2024-03-11T21:59:34.489745+0000","last_fullsized":"2024-03-11T21:59:34.489745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:06.466929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489253+0000","last_change":"2024-03-11T21:59:07.069411+0000","last_active":"2024-03-11T21:59:34.489253+0000","last_peered":"2024-03-11T21:59:34.489253+0000","last_clean":"2024-03-11T21:59:34.489253+0000","last_became_active":"2024-03-11T21:59:07.068720+0000","last_became_peered":"2024-03-11T21:59:07.068720+0000","last_unstale":"2024-03-11T21:59:34.489253+0000","last_undegraded":"2024-03-11T21:59:34.489253+0000","last_fullsized":"2024-03-11T21:59:34.489253+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:04:24.994863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486797+0000","last_change":"2024-03-11T21:59:03.040813+0000","last_active":"2024-03-11T21:59:34.486797+0000","last_peered":"2024-03-11T21:59:34.486797+0000","last_clean":"2024-03-11T21:59:34.486797+0000","last_became_active":"2024-03-11T21:59:03.040548+0000","last_became_peered":"2024-03-11T21:59:03.040548+0000","last_unstale":"2024-03-11T21:59:34.486797+0000","last_undegraded":"2024-03-11T21:59:34.486797+0000","last_fullsized":"2024-03-11T21:59:34.486797+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:14:27.751072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487230+0000","last_change":"2024-03-11T21:59:01.041126+0000","last_active":"2024-03-11T21:59:34.487230+0000","last_peered":"2024-03-11T21:59:34.487230+0000","last_clean":"2024-03-11T21:59:34.487230+0000","last_became_active":"2024-03-11T21:59:01.040771+0000","last_became_peered":"2024-03-11T21:59:01.040771+0000","last_unstale":"2024-03-11T21:59:34.487230+0000","last_undegraded":"2024-03-11T21:59:34.487230+0000","last_fullsized":"2024-03-11T21:59:34.487230+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:08:13.551981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489736+0000","last_change":"2024-03-11T21:59:05.056172+0000","last_active":"2024-03-11T21:59:34.489736+0000","last_peered":"2024-03-11T21:59:34.489736+0000","last_clean":"2024-03-11T21:59:34.489736+0000","last_became_active":"2024-03-11T21:59:05.055820+0000","last_became_peered":"2024-03-11T21:59:05.055820+0000","last_unstale":"2024-03-11T21:59:34.489736+0000","last_undegraded":"2024-03-11T21:59:34.489736+0000","last_fullsized":"2024-03-11T21:59:34.489736+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:36.140674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488209+0000","last_change":"2024-03-11T21:59:07.069036+0000","last_active":"2024-03-11T21:59:34.488209+0000","last_peered":"2024-03-11T21:59:34.488209+0000","last_clean":"2024-03-11T21:59:34.488209+0000","last_became_active":"2024-03-11T21:59:07.068719+0000","last_became_peered":"2024-03-11T21:59:07.068719+0000","last_unstale":"2024-03-11T21:59:34.488209+0000","last_undegraded":"2024-03-11T21:59:34.488209+0000","last_fullsized":"2024-03-11T21:59:34.488209+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:38:53.744781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489364+0000","last_change":"2024-03-11T21:59:01.038717+0000","last_active":"2024-03-11T21:59:34.489364+0000","last_peered":"2024-03-11T21:59:34.489364+0000","last_clean":"2024-03-11T21:59:34.489364+0000","last_became_active":"2024-03-11T21:59:01.038223+0000","last_became_peered":"2024-03-11T21:59:01.038223+0000","last_unstale":"2024-03-11T21:59:34.489364+0000","last_undegraded":"2024-03-11T21:59:34.489364+0000","last_fullsized":"2024-03-11T21:59:34.489364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:24.938122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087555+0000","last_change":"2024-03-11T21:59:03.042857+0000","last_active":"2024-03-11T21:59:36.087555+0000","last_peered":"2024-03-11T21:59:36.087555+0000","last_clean":"2024-03-11T21:59:36.087555+0000","last_became_active":"2024-03-11T21:59:03.042552+0000","last_became_peered":"2024-03-11T21:59:03.042552+0000","last_unstale":"2024-03-11T21:59:36.087555+0000","last_undegraded":"2024-03-11T21:59:36.087555+0000","last_fullsized":"2024-03-11T21:59:36.087555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:14:27.133202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087853+0000","last_change":"2024-03-11T21:59:05.056668+0000","last_active":"2024-03-11T21:59:36.087853+0000","last_peered":"2024-03-11T21:59:36.087853+0000","last_clean":"2024-03-11T21:59:36.087853+0000","last_became_active":"2024-03-11T21:59:05.056359+0000","last_became_peered":"2024-03-11T21:59:05.056359+0000","last_unstale":"2024-03-11T21:59:36.087853+0000","last_undegraded":"2024-03-11T21:59:36.087853+0000","last_fullsized":"2024-03-11T21:59:36.087853+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:43.694022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487333+0000","last_change":"2024-03-11T21:59:07.067965+0000","last_active":"2024-03-11T21:59:34.487333+0000","last_peered":"2024-03-11T21:59:34.487333+0000","last_clean":"2024-03-11T21:59:34.487333+0000","last_became_active":"2024-03-11T21:59:07.067509+0000","last_became_peered":"2024-03-11T21:59:07.067509+0000","last_unstale":"2024-03-11T21:59:34.487333+0000","last_undegraded":"2024-03-11T21:59:34.487333+0000","last_fullsized":"2024-03-11T21:59:34.487333+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:32:35.698812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488875+0000","last_change":"2024-03-11T21:59:01.038781+0000","last_active":"2024-03-11T21:59:34.488875+0000","last_peered":"2024-03-11T21:59:34.488875+0000","last_clean":"2024-03-11T21:59:34.488875+0000","last_became_active":"2024-03-11T21:59:01.037750+0000","last_became_peered":"2024-03-11T21:59:01.037750+0000","last_unstale":"2024-03-11T21:59:34.488875+0000","last_undegraded":"2024-03-11T21:59:34.488875+0000","last_fullsized":"2024-03-11T21:59:34.488875+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:50:19.762311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489496+0000","last_change":"2024-03-11T21:59:03.042122+0000","last_active":"2024-03-11T21:59:34.489496+0000","last_peered":"2024-03-11T21:59:34.489496+0000","last_clean":"2024-03-11T21:59:34.489496+0000","last_became_active":"2024-03-11T21:59:03.041816+0000","last_became_peered":"2024-03-11T21:59:03.041816+0000","last_unstale":"2024-03-11T21:59:34.489496+0000","last_undegraded":"2024-03-11T21:59:34.489496+0000","last_fullsized":"2024-03-11T21:59:34.489496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:30:05.746944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087574+0000","last_change":"2024-03-11T21:59:05.052923+0000","last_active":"2024-03-11T21:59:36.087574+0000","last_peered":"2024-03-11T21:59:36.087574+0000","last_clean":"2024-03-11T21:59:36.087574+0000","last_became_active":"2024-03-11T21:59:05.052521+0000","last_became_peered":"2024-03-11T21:59:05.052521+0000","last_unstale":"2024-03-11T21:59:36.087574+0000","last_undegraded":"2024-03-11T21:59:36.087574+0000","last_fullsized":"2024-03-11T21:59:36.087574+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:36:41.641848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489930+0000","last_change":"2024-03-11T21:59:07.075344+0000","last_active":"2024-03-11T21:59:34.489930+0000","last_peered":"2024-03-11T21:59:34.489930+0000","last_clean":"2024-03-11T21:59:34.489930+0000","last_became_active":"2024-03-11T21:59:07.075082+0000","last_became_peered":"2024-03-11T21:59:07.075082+0000","last_unstale":"2024-03-11T21:59:34.489930+0000","last_undegraded":"2024-03-11T21:59:34.489930+0000","last_fullsized":"2024-03-11T21:59:34.489930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:18.502772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489540+0000","last_change":"2024-03-11T21:59:01.037903+0000","last_active":"2024-03-11T21:59:34.489540+0000","last_peered":"2024-03-11T21:59:34.489540+0000","last_clean":"2024-03-11T21:59:34.489540+0000","last_became_active":"2024-03-11T21:59:01.036809+0000","last_became_peered":"2024-03-11T21:59:01.036809+0000","last_unstale":"2024-03-11T21:59:34.489540+0000","last_undegraded":"2024-03-11T21:59:34.489540+0000","last_fullsized":"2024-03-11T21:59:34.489540+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:37:15.249603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487956+0000","last_change":"2024-03-11T21:59:03.043631+0000","last_active":"2024-03-11T21:59:34.487956+0000","last_peered":"2024-03-11T21:59:34.487956+0000","last_clean":"2024-03-11T21:59:34.487956+0000","last_became_active":"2024-03-11T21:59:03.043341+0000","last_became_peered":"2024-03-11T21:59:03.043341+0000","last_unstale":"2024-03-11T21:59:34.487956+0000","last_undegraded":"2024-03-11T21:59:34.487956+0000","last_fullsized":"2024-03-11T21:59:34.487956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:57.638571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488035+0000","last_change":"2024-03-11T21:59:05.056062+0000","last_active":"2024-03-11T21:59:34.488035+0000","last_peered":"2024-03-11T21:59:34.488035+0000","last_clean":"2024-03-11T21:59:34.488035+0000","last_became_active":"2024-03-11T21:59:05.055570+0000","last_became_peered":"2024-03-11T21:59:05.055570+0000","last_unstale":"2024-03-11T21:59:34.488035+0000","last_undegraded":"2024-03-11T21:59:34.488035+0000","last_fullsized":"2024-03-11T21:59:34.488035+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:19:33.814384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489093+0000","last_change":"2024-03-11T21:59:07.068996+0000","last_active":"2024-03-11T21:59:34.489093+0000","last_peered":"2024-03-11T21:59:34.489093+0000","last_clean":"2024-03-11T21:59:34.489093+0000","last_became_active":"2024-03-11T21:59:07.068652+0000","last_became_peered":"2024-03-11T21:59:07.068652+0000","last_unstale":"2024-03-11T21:59:34.489093+0000","last_undegraded":"2024-03-11T21:59:34.489093+0000","last_fullsized":"2024-03-11T21:59:34.489093+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:16:25.487141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087758+0000","last_change":"2024-03-11T21:59:05.054058+0000","last_active":"2024-03-11T21:59:36.087758+0000","last_peered":"2024-03-11T21:59:36.087758+0000","last_clean":"2024-03-11T21:59:36.087758+0000","last_became_active":"2024-03-11T21:59:05.052901+0000","last_became_peered":"2024-03-11T21:59:05.052901+0000","last_unstale":"2024-03-11T21:59:36.087758+0000","last_undegraded":"2024-03-11T21:59:36.087758+0000","last_fullsized":"2024-03-11T21:59:36.087758+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:38.321381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487326+0000","last_change":"2024-03-11T21:59:01.041311+0000","last_active":"2024-03-11T21:59:34.487326+0000","last_peered":"2024-03-11T21:59:34.487326+0000","last_clean":"2024-03-11T21:59:34.487326+0000","last_became_active":"2024-03-11T21:59:01.040878+0000","last_became_peered":"2024-03-11T21:59:01.040878+0000","last_unstale":"2024-03-11T21:59:34.487326+0000","last_undegraded":"2024-03-11T21:59:34.487326+0000","last_fullsized":"2024-03-11T21:59:34.487326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:12.379926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490782+0000","last_change":"2024-03-11T21:59:03.045627+0000","last_active":"2024-03-11T21:59:34.490782+0000","last_peered":"2024-03-11T21:59:34.490782+0000","last_clean":"2024-03-11T21:59:34.490782+0000","last_became_active":"2024-03-11T21:59:03.045319+0000","last_became_peered":"2024-03-11T21:59:03.045319+0000","last_unstale":"2024-03-11T21:59:34.490782+0000","last_undegraded":"2024-03-11T21:59:34.490782+0000","last_fullsized":"2024-03-11T21:59:34.490782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:09.069520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489916+0000","last_change":"2024-03-11T21:59:07.069361+0000","last_active":"2024-03-11T21:59:34.489916+0000","last_peered":"2024-03-11T21:59:34.489916+0000","last_clean":"2024-03-11T21:59:34.489916+0000","last_became_active":"2024-03-11T21:59:07.068827+0000","last_became_peered":"2024-03-11T21:59:07.068827+0000","last_unstale":"2024-03-11T21:59:34.489916+0000","last_undegraded":"2024-03-11T21:59:34.489916+0000","last_fullsized":"2024-03-11T21:59:34.489916+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:07.505663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489066+0000","last_change":"2024-03-11T21:59:05.055222+0000","last_active":"2024-03-11T21:59:34.489066+0000","last_peered":"2024-03-11T21:59:34.489066+0000","last_clean":"2024-03-11T21:59:34.489066+0000","last_became_active":"2024-03-11T21:59:05.054668+0000","last_became_peered":"2024-03-11T21:59:05.054668+0000","last_unstale":"2024-03-11T21:59:34.489066+0000","last_undegraded":"2024-03-11T21:59:34.489066+0000","last_fullsized":"2024-03-11T21:59:34.489066+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:46:46.679656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486593+0000","last_change":"2024-03-11T21:59:01.039937+0000","last_active":"2024-03-11T21:59:34.486593+0000","last_peered":"2024-03-11T21:59:34.486593+0000","last_clean":"2024-03-11T21:59:34.486593+0000","last_became_active":"2024-03-11T21:59:01.039277+0000","last_became_peered":"2024-03-11T21:59:01.039277+0000","last_unstale":"2024-03-11T21:59:34.486593+0000","last_undegraded":"2024-03-11T21:59:34.486593+0000","last_fullsized":"2024-03-11T21:59:34.486593+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:05:27.755652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489534+0000","last_change":"2024-03-11T21:59:03.046380+0000","last_active":"2024-03-11T21:59:34.489534+0000","last_peered":"2024-03-11T21:59:34.489534+0000","last_clean":"2024-03-11T21:59:34.489534+0000","last_became_active":"2024-03-11T21:59:03.046051+0000","last_became_peered":"2024-03-11T21:59:03.046051+0000","last_unstale":"2024-03-11T21:59:34.489534+0000","last_undegraded":"2024-03-11T21:59:34.489534+0000","last_fullsized":"2024-03-11T21:59:34.489534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:08.018522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488908+0000","last_change":"2024-03-11T21:59:07.065874+0000","last_active":"2024-03-11T21:59:34.488908+0000","last_peered":"2024-03-11T21:59:34.488908+0000","last_clean":"2024-03-11T21:59:34.488908+0000","last_became_active":"2024-03-11T21:59:07.065633+0000","last_became_peered":"2024-03-11T21:59:07.065633+0000","last_unstale":"2024-03-11T21:59:34.488908+0000","last_undegraded":"2024-03-11T21:59:34.488908+0000","last_fullsized":"2024-03-11T21:59:34.488908+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:05:08.990245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489051+0000","last_change":"2024-03-11T21:59:01.035837+0000","last_active":"2024-03-11T21:59:34.489051+0000","last_peered":"2024-03-11T21:59:34.489051+0000","last_clean":"2024-03-11T21:59:34.489051+0000","last_became_active":"2024-03-11T21:59:01.035535+0000","last_became_peered":"2024-03-11T21:59:01.035535+0000","last_unstale":"2024-03-11T21:59:34.489051+0000","last_undegraded":"2024-03-11T21:59:34.489051+0000","last_fullsized":"2024-03-11T21:59:34.489051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:25:41.324819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488165+0000","last_change":"2024-03-11T21:59:03.045158+0000","last_active":"2024-03-11T21:59:34.488165+0000","last_peered":"2024-03-11T21:59:34.488165+0000","last_clean":"2024-03-11T21:59:34.488165+0000","last_became_active":"2024-03-11T21:59:03.044683+0000","last_became_peered":"2024-03-11T21:59:03.044683+0000","last_unstale":"2024-03-11T21:59:34.488165+0000","last_undegraded":"2024-03-11T21:59:34.488165+0000","last_fullsized":"2024-03-11T21:59:34.488165+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:52:39.490851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488562+0000","last_change":"2024-03-11T21:59:05.052899+0000","last_active":"2024-03-11T21:59:34.488562+0000","last_peered":"2024-03-11T21:59:34.488562+0000","last_clean":"2024-03-11T21:59:34.488562+0000","last_became_active":"2024-03-11T21:59:05.052542+0000","last_became_peered":"2024-03-11T21:59:05.052542+0000","last_unstale":"2024-03-11T21:59:34.488562+0000","last_undegraded":"2024-03-11T21:59:34.488562+0000","last_fullsized":"2024-03-11T21:59:34.488562+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:42.009843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487184+0000","last_change":"2024-03-11T21:59:07.078406+0000","last_active":"2024-03-11T21:59:34.487184+0000","last_peered":"2024-03-11T21:59:34.487184+0000","last_clean":"2024-03-11T21:59:34.487184+0000","last_became_active":"2024-03-11T21:59:07.078044+0000","last_became_peered":"2024-03-11T21:59:07.078044+0000","last_unstale":"2024-03-11T21:59:34.487184+0000","last_undegraded":"2024-03-11T21:59:34.487184+0000","last_fullsized":"2024-03-11T21:59:34.487184+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:27.192333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:36.087765+0000","last_change":"2024-03-11T21:59:05.054810+0000","last_active":"2024-03-11T21:59:36.087765+0000","last_peered":"2024-03-11T21:59:36.087765+0000","last_clean":"2024-03-11T21:59:36.087765+0000","last_became_active":"2024-03-11T21:59:05.054482+0000","last_became_peered":"2024-03-11T21:59:05.054482+0000","last_unstale":"2024-03-11T21:59:36.087765+0000","last_undegraded":"2024-03-11T21:59:36.087765+0000","last_fullsized":"2024-03-11T21:59:36.087765+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:57:26.178250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488958+0000","last_change":"2024-03-11T21:59:01.035916+0000","last_active":"2024-03-11T21:59:34.488958+0000","last_peered":"2024-03-11T21:59:34.488958+0000","last_clean":"2024-03-11T21:59:34.488958+0000","last_became_active":"2024-03-11T21:59:01.035489+0000","last_became_peered":"2024-03-11T21:59:01.035489+0000","last_unstale":"2024-03-11T21:59:34.488958+0000","last_undegraded":"2024-03-11T21:59:34.488958+0000","last_fullsized":"2024-03-11T21:59:34.488958+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:57:46.263494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488387+0000","last_change":"2024-03-11T21:59:03.042161+0000","last_active":"2024-03-11T21:59:34.488387+0000","last_peered":"2024-03-11T21:59:34.488387+0000","last_clean":"2024-03-11T21:59:34.488387+0000","last_became_active":"2024-03-11T21:59:03.041755+0000","last_became_peered":"2024-03-11T21:59:03.041755+0000","last_unstale":"2024-03-11T21:59:34.488387+0000","last_undegraded":"2024-03-11T21:59:34.488387+0000","last_fullsized":"2024-03-11T21:59:34.488387+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:18:25.127890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488395+0000","last_change":"2024-03-11T21:59:07.067618+0000","last_active":"2024-03-11T21:59:34.488395+0000","last_peered":"2024-03-11T21:59:34.488395+0000","last_clean":"2024-03-11T21:59:34.488395+0000","last_became_active":"2024-03-11T21:59:07.067318+0000","last_became_peered":"2024-03-11T21:59:07.067318+0000","last_unstale":"2024-03-11T21:59:34.488395+0000","last_undegraded":"2024-03-11T21:59:34.488395+0000","last_fullsized":"2024-03-11T21:59:34.488395+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:44.518080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.491104+0000","last_change":"2024-03-11T21:59:01.039025+0000","last_active":"2024-03-11T21:59:34.491104+0000","last_peered":"2024-03-11T21:59:34.491104+0000","last_clean":"2024-03-11T21:59:34.491104+0000","last_became_active":"2024-03-11T21:59:01.038681+0000","last_became_peered":"2024-03-11T21:59:01.038681+0000","last_unstale":"2024-03-11T21:59:34.491104+0000","last_undegraded":"2024-03-11T21:59:34.491104+0000","last_fullsized":"2024-03-11T21:59:34.491104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:00.822840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.490013+0000","last_change":"2024-03-11T21:59:03.042063+0000","last_active":"2024-03-11T21:59:34.490013+0000","last_peered":"2024-03-11T21:59:34.490013+0000","last_clean":"2024-03-11T21:59:34.490013+0000","last_became_active":"2024-03-11T21:59:03.041612+0000","last_became_peered":"2024-03-11T21:59:03.041612+0000","last_unstale":"2024-03-11T21:59:34.490013+0000","last_undegraded":"2024-03-11T21:59:34.490013+0000","last_fullsized":"2024-03-11T21:59:34.490013+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:55.201857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488088+0000","last_change":"2024-03-11T21:59:05.055023+0000","last_active":"2024-03-11T21:59:34.488088+0000","last_peered":"2024-03-11T21:59:34.488088+0000","last_clean":"2024-03-11T21:59:34.488088+0000","last_became_active":"2024-03-11T21:59:05.054466+0000","last_became_peered":"2024-03-11T21:59:05.054466+0000","last_unstale":"2024-03-11T21:59:34.488088+0000","last_undegraded":"2024-03-11T21:59:34.488088+0000","last_fullsized":"2024-03-11T21:59:34.488088+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:26:08.514384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.486366+0000","last_change":"2024-03-11T21:59:07.077441+0000","last_active":"2024-03-11T21:59:34.486366+0000","last_peered":"2024-03-11T21:59:34.486366+0000","last_clean":"2024-03-11T21:59:34.486366+0000","last_became_active":"2024-03-11T21:59:07.077036+0000","last_became_peered":"2024-03-11T21:59:07.077036+0000","last_unstale":"2024-03-11T21:59:34.486366+0000","last_undegraded":"2024-03-11T21:59:34.486366+0000","last_fullsized":"2024-03-11T21:59:34.486366+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:33:20.121043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489222+0000","last_change":"2024-03-11T21:59:01.038981+0000","last_active":"2024-03-11T21:59:34.489222+0000","last_peered":"2024-03-11T21:59:34.489222+0000","last_clean":"2024-03-11T21:59:34.489222+0000","last_became_active":"2024-03-11T21:59:01.038414+0000","last_became_peered":"2024-03-11T21:59:01.038414+0000","last_unstale":"2024-03-11T21:59:34.489222+0000","last_undegraded":"2024-03-11T21:59:34.489222+0000","last_fullsized":"2024-03-11T21:59:34.489222+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:24.519203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488696+0000","last_change":"2024-03-11T21:59:03.042154+0000","last_active":"2024-03-11T21:59:34.488696+0000","last_peered":"2024-03-11T21:59:34.488696+0000","last_clean":"2024-03-11T21:59:34.488696+0000","last_became_active":"2024-03-11T21:59:03.041645+0000","last_became_peered":"2024-03-11T21:59:03.041645+0000","last_unstale":"2024-03-11T21:59:34.488696+0000","last_undegraded":"2024-03-11T21:59:34.488696+0000","last_fullsized":"2024-03-11T21:59:34.488696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:12:26.101680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487378+0000","last_change":"2024-03-11T21:59:05.054428+0000","last_active":"2024-03-11T21:59:34.487378+0000","last_peered":"2024-03-11T21:59:34.487378+0000","last_clean":"2024-03-11T21:59:34.487378+0000","last_became_active":"2024-03-11T21:59:05.053502+0000","last_became_peered":"2024-03-11T21:59:05.053502+0000","last_unstale":"2024-03-11T21:59:34.487378+0000","last_undegraded":"2024-03-11T21:59:34.487378+0000","last_fullsized":"2024-03-11T21:59:34.487378+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:19:55.095498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.487405+0000","last_change":"2024-03-11T21:59:07.067728+0000","last_active":"2024-03-11T21:59:34.487405+0000","last_peered":"2024-03-11T21:59:34.487405+0000","last_clean":"2024-03-11T21:59:34.487405+0000","last_became_active":"2024-03-11T21:59:07.067145+0000","last_became_peered":"2024-03-11T21:59:07.067145+0000","last_unstale":"2024-03-11T21:59:34.487405+0000","last_undegraded":"2024-03-11T21:59:34.487405+0000","last_fullsized":"2024-03-11T21:59:34.487405+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:06.043044+0000","last_clean_scrub_stamp":"2024-03-11T21:59:06.043044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:08.065774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.488232+0000","last_change":"2024-03-11T21:59:01.037676+0000","last_active":"2024-03-11T21:59:34.488232+0000","last_peered":"2024-03-11T21:59:34.488232+0000","last_clean":"2024-03-11T21:59:34.488232+0000","last_became_active":"2024-03-11T21:59:01.037380+0000","last_became_peered":"2024-03-11T21:59:01.037380+0000","last_unstale":"2024-03-11T21:59:34.488232+0000","last_undegraded":"2024-03-11T21:59:34.488232+0000","last_fullsized":"2024-03-11T21:59:34.488232+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:00.015944+0000","last_clean_scrub_stamp":"2024-03-11T21:59:00.015944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:17.909275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489646+0000","last_change":"2024-03-11T21:59:03.042871+0000","last_active":"2024-03-11T21:59:34.489646+0000","last_peered":"2024-03-11T21:59:34.489646+0000","last_clean":"2024-03-11T21:59:34.489646+0000","last_became_active":"2024-03-11T21:59:03.042446+0000","last_became_peered":"2024-03-11T21:59:03.042446+0000","last_unstale":"2024-03-11T21:59:34.489646+0000","last_undegraded":"2024-03-11T21:59:34.489646+0000","last_fullsized":"2024-03-11T21:59:34.489646+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:02.024676+0000","last_clean_scrub_stamp":"2024-03-11T21:59:02.024676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:26:04.488628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:59:34.489751+0000","last_change":"2024-03-11T21:59:05.051540+0000","last_active":"2024-03-11T21:59:34.489751+0000","last_peered":"2024-03-11T21:59:34.489751+0000","last_clean":"2024-03-11T21:59:34.489751+0000","last_became_active":"2024-03-11T21:59:05.051219+0000","last_became_peered":"2024-03-11T21:59:05.051219+0000","last_unstale":"2024-03-11T21:59:34.489751+0000","last_undegraded":"2024-03-11T21:59:34.489751+0000","last_fullsized":"2024-03-11T21:59:34.489751+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:59:04.032237+0000","last_clean_scrub_stamp":"2024-03-11T21:59:04.032237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:27.734526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411342,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:59:49.999 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:59:49.999 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T21:59:49.999 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T21:59:49.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T21:59:50.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:50 smithi151 ceph-mon[25773]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:50.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:50 smithi151 ceph-mon[25773]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:59:50.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:50 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:59:50.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:50 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ::ffff:172.21.15.176 - - [11/Mar/2024:21:59:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:59:50.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:50 smithi176 ceph-mon[27622]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:50.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:50 smithi176 ceph-mon[27622]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:59:50.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:50 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:59:50.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:50 smithi189 ceph-mon[27727]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:50.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:50 smithi189 ceph-mon[27727]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:59:50.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:50 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:59:50.810 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:50.938 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 21:59:50 smithi151 systemd[1]: Started Ceph alertmanager.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T21:59:51.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:51] ENGINE Bus STOPPING 2024-03-11T21:59:51.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:59:51.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:51] ENGINE Bus STOPPED 2024-03-11T21:59:51.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:51] ENGINE Bus STARTING 2024-03-11T21:59:51.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:51] ENGINE Serving on http://:::9283 2024-03-11T21:59:51.304 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:21:59:51] ENGINE Bus STARTED 2024-03-11T21:59:52.048 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:52.048 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.048 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.048 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.048 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:51 smithi189 ceph-mon[27727]: Deploying daemon grafana.a on smithi176 2024-03-11T21:59:52.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:52.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:51 smithi151 ceph-mon[25773]: Deploying daemon grafana.a on smithi176 2024-03-11T21:59:52.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:52.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:52.100 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:51 smithi176 ceph-mon[27622]: Deploying daemon grafana.a on smithi176 2024-03-11T21:59:52.869 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 21:59:52 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a[48891]: ts=2024-03-11T21:59:52.690Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000681668s 2024-03-11T21:59:54.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:53 smithi151 ceph-mon[25773]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:54.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:53 smithi176 ceph-mon[27622]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:54.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:53 smithi189 ceph-mon[27727]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:59:54.157 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-11T21:59:54.158 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T21:59:54.700 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T21:59:54.700 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T21:59:54.700 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T21:59:54.711 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi151.front.sepia.ceph.com 2024-03-11T21:59:54.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T21:59:55.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:54 smithi151 ceph-mon[25773]: from='client.? 172.21.15.151:0/1639149577' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:59:55.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:54 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:55.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:54 smithi176 ceph-mon[27622]: from='client.? 172.21.15.151:0/1639149577' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:59:55.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:54 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:55.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:54 smithi189 ceph-mon[27727]: from='client.? 172.21.15.151:0/1639149577' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:59:55.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:54 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T21:59:55.496 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:59:55.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:55 smithi151 ceph-mon[25773]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:55.986 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:55 smithi176 ceph-mon[27622]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:56.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:55 smithi189 ceph-mon[27727]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:58.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:57 smithi151 ceph-mon[25773]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:58.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:57 smithi176 ceph-mon[27622]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:58.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:57 smithi189 ceph-mon[27727]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:59:58.236 INFO:teuthology.orchestra.run.smithi151.stdout:167 167 2024-03-11T21:59:58.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T21:59:59.665 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:00.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 21:59:59 smithi151 ceph-mon[25773]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:00.099 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 21:59:59 smithi176 ceph-mon[27622]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:00.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 21:59:59 smithi189 ceph-mon[27727]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:00.686 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:00 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ::ffff:172.21.15.176 - - [11/Mar/2024:22:00:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:00:00.963 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:00 smithi151 ceph-mon[25773]: overall HEALTH_OK 2024-03-11T22:00:00.963 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:00:00 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a[48891]: ts=2024-03-11T22:00:00.692Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003259176s 2024-03-11T22:00:01.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:00 smithi189 ceph-mon[27727]: overall HEALTH_OK 2024-03-11T22:00:01.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:00 smithi176 ceph-mon[27622]: overall HEALTH_OK 2024-03-11T22:00:01.808 INFO:teuthology.orchestra.run.smithi151.stdout:Backend: cephadm 2024-03-11T22:00:01.809 INFO:teuthology.orchestra.run.smithi151.stdout:Available: Yes 2024-03-11T22:00:01.809 INFO:teuthology.orchestra.run.smithi151.stdout:Paused: No 2024-03-11T22:00:02.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:01 smithi151 ceph-mon[25773]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:02.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:01 smithi189 ceph-mon[27727]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:02.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:01 smithi176 ceph-mon[27622]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:02.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T22:00:02.644 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:03.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:02 smithi151 ceph-mon[25773]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:03.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:02 smithi189 ceph-mon[27727]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:03.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:02 smithi176 ceph-mon[27622]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:04.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:03 smithi151 ceph-mon[25773]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:04.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:03 smithi189 ceph-mon[27727]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:04.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:03 smithi176 ceph-mon[27622]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:agent.smithi151 smithi151 running 40s ago 3m - - 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:agent.smithi176 smithi176 running 56s ago 3m - - 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:agent.smithi189 smithi189 running 55s ago 3m - - 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager.a smithi151 *:9093,9094 starting - - - - 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.x smithi176 *:8443,8765 running (2m) 56s ago 2m 451M - 19.0.0-1607-g94a53970 a6477a474d71 c86fd27ac212 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.y smithi151 *:9283,8765 running (4m) 40s ago 4m 511M - 19.0.0-1607-g94a53970 a6477a474d71 6348834d01f9 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:mgr.z smithi189 *:8443,8765 running (2m) 55s ago 2m 449M - 19.0.0-1607-g94a53970 a6477a474d71 79be1823948d 2024-03-11T22:00:04.480 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a smithi151 running (4m) 40s ago 4m 46.7M 2048M 19.0.0-1607-g94a53970 a6477a474d71 d72ded8e46bc 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b smithi176 running (2m) 56s ago 2m 43.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 3fcc03fa3228 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c smithi189 running (3m) 55s ago 3m 41.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 69d0055a18d1 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 smithi151 running (2m) 40s ago 2m 57.6M 4096M 19.0.0-1607-g94a53970 a6477a474d71 66c24425769c 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 smithi176 running (114s) 56s ago 113s 54.1M 4096M 19.0.0-1607-g94a53970 a6477a474d71 c9106bbde9c9 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 smithi189 running (87s) 55s ago 86s 51.4M 4096M 19.0.0-1607-g94a53970 a6477a474d71 1ca0bef246f3 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus.a smithi176 *:9095 starting - - - - 2024-03-11T22:00:04.481 INFO:teuthology.orchestra.run.smithi151.stdout:rgw.foo.a smithi151 *:80 running (65s) 40s ago 65s 77.1M - 19.0.0-1607-g94a53970 a6477a474d71 b2d47ebb63d9 2024-03-11T22:00:05.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:04 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:00:05.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T22:00:05.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:04 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:00:05.168 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:04 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:00:05.324 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:06.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:05 smithi151 ceph-mon[25773]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:06.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:05 smithi151 ceph-mon[25773]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:06.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:05 smithi189 ceph-mon[27727]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:06.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:05 smithi189 ceph-mon[27727]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:06.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:05 smithi176 ceph-mon[27622]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:06.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:05 smithi176 ceph-mon[27622]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:07.116 INFO:teuthology.orchestra.run.smithi151.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T22:00:07.116 INFO:teuthology.orchestra.run.smithi151.stdout:agent 3/3 59s ago 32s * 2024-03-11T22:00:07.116 INFO:teuthology.orchestra.run.smithi151.stdout:alertmanager ?:9093,9094 0/1 - 63s smithi151=a;count:1 2024-03-11T22:00:07.116 INFO:teuthology.orchestra.run.smithi151.stdout:grafana ?:3000 0/1 - 60s smithi176=a;count:1 2024-03-11T22:00:07.116 INFO:teuthology.orchestra.run.smithi151.stdout:mgr 3/3 59s ago 2m smithi151=y;smithi176=x;smithi189=z;count:3 2024-03-11T22:00:07.116 INFO:teuthology.orchestra.run.smithi151.stdout:mon 3/3 59s ago 3m smithi151:172.21.15.151=a;smithi176:172.21.15.176=b;smithi189:172.21.15.189=c;count:3 2024-03-11T22:00:07.117 INFO:teuthology.orchestra.run.smithi151.stdout:node-exporter ?:9100 0/3 - 65s smithi151=a;smithi176=b;smithi189=c;count:3 2024-03-11T22:00:07.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd 3 59s ago - 2024-03-11T22:00:07.117 INFO:teuthology.orchestra.run.smithi151.stdout:prometheus ?:9095 0/1 - 68s smithi176=a;count:1 2024-03-11T22:00:07.117 INFO:teuthology.orchestra.run.smithi151.stdout:rgw.foo.a ?:80 1/1 43s ago 67s smithi151=foo.a;count:1 2024-03-11T22:00:07.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T22:00:07.960 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:08.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:07 smithi151 ceph-mon[25773]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:08.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:07 smithi151 ceph-mon[25773]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:08.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:07 smithi189 ceph-mon[27727]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:08.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:07 smithi189 ceph-mon[27727]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:07 smithi176 ceph-mon[27622]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:08.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:07 smithi176 ceph-mon[27622]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:09.746 INFO:teuthology.orchestra.run.smithi151.stdout:HOST ADDR LABELS STATUS 2024-03-11T22:00:09.747 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 172.21.15.151 2024-03-11T22:00:09.747 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 172.21.15.176 2024-03-11T22:00:09.747 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 172.21.15.189 2024-03-11T22:00:09.747 INFO:teuthology.orchestra.run.smithi151.stdout:3 hosts in cluster 2024-03-11T22:00:10.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:09 smithi151 ceph-mon[25773]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:10.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:09 smithi176 ceph-mon[27622]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:10.127 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:09 smithi189 ceph-mon[27727]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:10.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T22:00:10.599 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:10.720 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:10 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ::ffff:172.21.15.176 - - [11/Mar/2024:22:00:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:00:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:10 smithi176 ceph-mon[27622]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:11.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:10 smithi151 ceph-mon[25773]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:11.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:10 smithi189 ceph-mon[27727]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:11.599 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:11 smithi176 systemd[1]: Starting Ceph grafana.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:11.948 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:11 smithi176 ceph-mon[27622]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:11.949 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:11 smithi176 podman[38345]: 2024-03-11 22:00:11.654855674 +0000 UTC m=+0.114231663 container create 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, architecture=x86_64, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=ubi8, io.openshift.expose-services=, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public) 2024-03-11T22:00:11.949 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:11 smithi176 podman[38345]: 2024-03-11 22:00:11.558358588 +0000 UTC m=+0.017734513 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T22:00:11.949 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:11 smithi176 podman[38345]: 2024-03-11 22:00:11.955134277 +0000 UTC m=+0.414510205 container init 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.openshift.expose-services=, description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, name=ubi8, distribution-scope=public, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.tags=base rhel8) 2024-03-11T22:00:11.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:11 smithi151 ceph-mon[25773]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:11 smithi176 podman[38345]: 2024-03-11 22:00:11.961075912 +0000 UTC m=+0.420451832 container start 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-type=git, io.buildah.version=1.32.0) 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 bash[38345]: 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 systemd[1]: Started Ceph grafana.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114228208Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.11442821Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114447417Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114454491Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114460466Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.11446669Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.1144724Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T22:00:12.200 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114482272Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T22:00:12.201 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114488157Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T22:00:12.201 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114493751Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T22:00:12.201 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114499122Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114504377Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114509916Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=settings t=2024-03-11T22:00:12.114515539Z level=info msg="App mode production" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=sqlstore t=2024-03-11T22:00:12.11455775Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=sqlstore t=2024-03-11T22:00:12.114573298Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.115471338Z level=info msg="Starting DB migrations" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.115554444Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.116074557Z level=info msg="Executing migration" id="create user table" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.116514563Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.117345266Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.1181923Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T22:00:12.202 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.119024822Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.119880167Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.120743264Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.123413832Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.123833639Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.126195082Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.128755392Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.129305717Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.129722043Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.130003986Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.130436725Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.130799596Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.131226223Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.131624855Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T22:00:12.203 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.132008701Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.133122205Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.133557128Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.134208276Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.134615061Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.135467375Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.136307241Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.136657354Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.137447258Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.13783907Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.138261182Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.139050756Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.139865426Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.140706152Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.141557732Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T22:00:12.204 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.142381124Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.143163498Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.144031623Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.144781664Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.145168553Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.145958113Z level=info msg="Executing migration" id="create star table" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.146738882Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.147547682Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.14799534Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.148826316Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.149672703Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.150114125Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.150965876Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.151417565Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.152195527Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T22:00:12.205 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.152502424Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.152791261Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.153197569Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.153599803Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.154465597Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.155235071Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.155646797Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.156462199Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.157425236Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.15782594Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.158658967Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.161555437Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.164324851Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.167038511Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.169639237Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T22:00:12.206 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.170213838Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.173284249Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.175989448Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.178582148Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.179105165Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.182113745Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.18510345Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.187542203Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.187856287Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.190631918Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.191173172Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.19404716Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.194542516Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.194833614Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.195260633Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.197703369Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.200355516Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T22:00:12.207 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.20313735Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T22:00:12.208 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.203591973Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T22:00:12.208 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.20632327Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T22:00:12.294 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:11 smithi189 ceph-mon[27727]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:12.451 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.208965371Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T22:00:12.451 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.211656224Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T22:00:12.451 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.212108779Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T22:00:12.451 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.214817046Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T22:00:12.451 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.217527375Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.220188423Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.22065473Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.223320952Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.22368812Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.226131452Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.22873644Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.22922889Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.2300345Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.230474237Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.231371303Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.231968125Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.233009579Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.233811638Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.234663114Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T22:00:12.452 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.235470655Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.235852177Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.236715019Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.237646881Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.238364061Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.238723433Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.23953851Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.240422963Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.241163634Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.241854799Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.242549843Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.243355667Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.243814552Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.24426364Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T22:00:12.453 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.244713305Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T22:00:12.454 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.245802021Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T22:00:12.454 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.24643319Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T22:00:12.454 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.247273Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T22:00:12.454 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.248270776Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.248983834Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.250040595Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.250456525Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.250876396Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.251344946Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.25177195Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.252149517Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.252524748Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.252805326Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.253366978Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T22:00:12.455 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.253910705Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.254252064Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.254782048Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.255352985Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.255753874Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.25616656Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.256594301Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.257020979Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.257456395Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.257875011Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.258197149Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.258488738Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.259042445Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.259611103Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.259914918Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T22:00:12.456 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.260335134Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.260803138Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.261641472Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.262579785Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.263484727Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.264204902Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.267251384Z level=info msg="Executing migration" id="create session table" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.270004812Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.272816728Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.275789167Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.278542761Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.28156576Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.284207781Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.286924567Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.289735229Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T22:00:12.457 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.292765969Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.295659106Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.298318227Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.301145051Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.303879313Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.306698575Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.309439652Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.312391791Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.315194194Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.317735024Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.320528523Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.323172475Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.325922446Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T22:00:12.458 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.329329148Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.331471111Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.334114574Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.336669881Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.339463842Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.342022919Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.344675481Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.347175911Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.3500427Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.352589445Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.355205709Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.357798056Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.360567025Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.361283Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.362241689Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T22:00:12.459 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.362987159Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.363973577Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.364797259Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.365510119Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.366427332Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.366986134Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.367405131Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.36823919Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.368917212Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.36924057Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.370700063Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.371263322Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.372110977Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.373014301Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.373463196Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T22:00:12.460 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.373866732Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.374746222Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.375634225Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.376577427Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.377368606Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.378410056Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.379221328Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.380176739Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.380967233Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.381873339Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.382738765Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.383595986Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.384449579Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.385287712Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.386155617Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T22:00:12.461 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.387711036Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.388607184Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.389614286Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.392345399Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.395224628Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.398256935Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.401155542Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.404007946Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.406895491Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.409697444Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.412533683Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.415296457Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.418231296Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.421011066Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T22:00:12.462 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.423637982Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.426617301Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.429338426Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.432318088Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.435061722Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.438090528Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.440755033Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.443546529Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.446328774Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.449066959Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.451841521Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.454400968Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.45724922Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T22:00:12.463 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.459998889Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T22:00:12.703 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.46275015Z level=info msg="Executing migration" id="create team table" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.463494415Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.466316561Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.468923093Z level=info msg="Executing migration" id="create team member table" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.471444919Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.471962616Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.474665767Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.477312894Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.480048175Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.480742751Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.481605754Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.48200553Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T22:00:12.704 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.482506495Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T22:00:12.705 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.483382838Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T22:00:12.705 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.483834647Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T22:00:12.705 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.484707177Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T22:00:12.705 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.485663859Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T22:00:12.705 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.486531714Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T22:00:12.705 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.4872477Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.488093785Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.489150987Z level=info msg="Executing migration" id="create tag table" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.489791466Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.490663619Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.491508023Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.492354211Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.49319495Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T22:00:12.706 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.495042557Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.495947404Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.496822607Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.497709947Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.498215554Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.498724151Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.499313341Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.499664582Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.500482152Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.50140772Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.502704831Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.503504126Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.503952272Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.504702003Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T22:00:12.707 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.505097909Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.505523374Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.505941641Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.50674524Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.507183506Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.507606062Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.508360571Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.508764323Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.509185355Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.509593333Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.510428319Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.510713749Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.510985692Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.511405803Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.511821371Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T22:00:12.708 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.51226007Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.512557914Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.512969996Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.513403537Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.514169962Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.514596165Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.517075319Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.517532484Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.51781557Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.520402928Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.523022832Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.525808179Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.528344346Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.528875936Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T22:00:12.709 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.532005694Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.53464609Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.537764564Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.53855319Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.538984208Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.541825083Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.545183087Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.548573404Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.54907503Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.552044292Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.55474706Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.555403177Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.558183665Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.561031847Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T22:00:12.710 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.563867459Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.56656585Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.569298371Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.572050359Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.574822464Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.577551029Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.580056854Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.582859125Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.585434619Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.588109505Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.590941625Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.593594238Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.596048759Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.598914251Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T22:00:12.711 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.601545436Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.604128508Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.604429163Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.607201542Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.609895753Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.612728959Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.615356601Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.617854955Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.618709483Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.619740351Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.620708401Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.621509947Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T22:00:12.712 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.622486705Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T22:00:12.713 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.623304789Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T22:00:12.713 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.624193921Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T22:00:12.713 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.625002246Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T22:00:12.713 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.625844412Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T22:00:12.713 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.626750162Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.627672776Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.628429623Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.6292313Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.630083088Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.631022468Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.631697436Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.632304593Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.63242278Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.633084391Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.633952498Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.63481429Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.635004368Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T22:00:12.714 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.635182599Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.635741491Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.636542003Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.637381151Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.638313835Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.639116759Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.640002978Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.64083703Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.64163432Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.642536492Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.643385433Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.647766436Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.648683063Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.649533507Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.653818645Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T22:00:12.715 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.658036341Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.658770675Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.659206614Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.659548008Z level=info msg="Executing migration" id="create permission table" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.659933987Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.660356493Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.660798079Z level=info msg="Executing migration" id="create role table" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.661188793Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.662087801Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.662996278Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.663429808Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.663843296Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.66426414Z level=info msg="Executing migration" id="create team role table" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.664643708Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T22:00:12.716 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.665063073Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.665507813Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.665916528Z level=info msg="Executing migration" id="create user role table" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.666342616Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.666791885Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.667249608Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.667698945Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.668161372Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.668923742Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.669784959Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.670814838Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.671669037Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.672514433Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.673382947Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.674235441Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T22:00:12.717 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.675101479Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.675985932Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.676971549Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.677820396Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.678738997Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.679573855Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.680465936Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.681330865Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.682152832Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.683034725Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.683914411Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.684786974Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.685655486Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.686683825Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T22:00:12.718 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.687528778Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.68904101Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.689527434Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.69244457Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.695359746Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.698154745Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.701098274Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.701208075Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.703867146Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.703978454Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.706769707Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.709720078Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T22:00:12.719 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.712557134Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.715410164Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.71832208Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.72118368Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.723983604Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.726854857Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.729630793Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.732560796Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.735510635Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.738450845Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T22:00:13.100 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.741743867Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.744445016Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.747081706Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.749967474Z level=info msg="Executing migration" id="add share column" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.752568365Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.75530256Z level=info msg="Executing migration" id="create file table" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.757845835Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.758421166Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.761109171Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T22:00:13.101 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.763836149Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.766400634Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.766886955Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.769587513Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.772138334Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.774789441Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.776318085Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.778922206Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.781693119Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.782531924Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.782973124Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.783866133Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.785021004Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.786034167Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T22:00:13.102 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.786877593Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.797012988Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.797820058Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.798662228Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.802323183Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.803137519Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.803993127Z level=info msg="Executing migration" id="create folder table" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.80487518Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.805685977Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.806536833Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.807393856Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=migrator t=2024-03-11T22:00:12.80823954Z level=info msg="migrations completed" performed=485 skipped=0 duration=692.714137ms 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=sqlstore t=2024-03-11T22:00:12.808649327Z level=info msg="Created default organization" 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=plugin.loader t=2024-03-11T22:00:12.880487169Z level=info msg="Plugin registered" pluginID=input 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=plugin.loader t=2024-03-11T22:00:12.910027097Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=plugin.loader t=2024-03-11T22:00:12.910066931Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T22:00:13.103 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=secrets t=2024-03-11T22:00:12.910319223Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T22:00:13.104 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=query_data t=2024-03-11T22:00:12.91456556Z level=info msg="Query Service initialization" 2024-03-11T22:00:13.104 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:12 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=live.push_http t=2024-03-11T22:00:12.917300379Z level=info msg="Live Push Gateway initialization" 2024-03-11T22:00:13.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:13 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:13 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:13 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:13 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:13 smithi151 ceph-mon[25773]: Deploying daemon node-exporter.a on smithi151 2024-03-11T22:00:13.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:13 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:13 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:13 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:13 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.378 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:13 smithi189 ceph-mon[27727]: Deploying daemon node-exporter.a on smithi151 2024-03-11T22:00:13.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:13.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-mon[27622]: Deploying daemon node-exporter.a on smithi151 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JJ400BGN 372G No 49s ago Has a FileSystem, LVM detected 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/nvme1n1 ssd Linux_4c7c7e7b0eb6023c53b1 89.4G Yes 49s ago 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/nvme2n1 ssd Linux_515856c017f0909c15a7 89.4G Yes 49s ago 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/nvme3n1 ssd Linux_4413c8b1878e15261397 89.4G Yes 49s ago 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/nvme4n1 ssd Linux_22ae7e8451852b8aba00 89.4G No 49s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XK16 931G No 49s ago Has partitions 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330099400BGN 372G No 65s ago Has a FileSystem, LVM detected 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 /dev/nvme1n1 ssd Linux_3419fd1895239dbce5f3 89.4G Yes 65s ago 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 /dev/nvme2n1 ssd Linux_e2a76087d578020c7e93 89.4G Yes 65s ago 2024-03-11T22:00:13.621 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 /dev/nvme3n1 ssd Linux_48fe081f6ae86fd6b4c6 89.4G Yes 65s ago 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 /dev/nvme4n1 ssd Linux_5c0009d97f35ee05ff2f 89.4G No 65s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi176 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XKB1 931G No 65s ago Has partitions 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 84s ago Has a FileSystem, LVM detected 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 /dev/nvme1n1 ssd Linux_bc90b600a98798b48ce1 89.4G Yes 84s ago 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 /dev/nvme2n1 ssd Linux_e77c53556cf9c79f9bf4 89.4G Yes 84s ago 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 /dev/nvme3n1 ssd Linux_763fa3016cae2b9de0e8 89.4G Yes 84s ago 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 /dev/nvme4n1 ssd Linux_caf5c065b726d7e361b3 89.4G No 84s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:00:13.622 INFO:teuthology.orchestra.run.smithi151.stdout:smithi189 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XKLY 931G No 84s ago Has partitions 2024-03-11T22:00:13.849 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=infra.usagestats.collector t=2024-03-11T22:00:13.585003565Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T22:00:13.849 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=provisioning.datasources t=2024-03-11T22:00:13.58616065Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T22:00:13.849 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=provisioning.datasources t=2024-03-11T22:00:13.597295929Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=provisioning.alerting t=2024-03-11T22:00:13.608258006Z level=info msg="starting to provision alerting" 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=provisioning.alerting t=2024-03-11T22:00:13.608276539Z level=info msg="finished to provision alerting" 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=ngalert.state.manager t=2024-03-11T22:00:13.608353621Z level=info msg="Warming state cache for startup" 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=ngalert.state.manager t=2024-03-11T22:00:13.608503471Z level=info msg="State cache has been initialized" states=0 duration=149.197µs 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=ticker t=2024-03-11T22:00:13.608567501Z level=info msg=starting first_tick=2024-03-11T22:00:20Z 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=ngalert.multiorg.alertmanager t=2024-03-11T22:00:13.608590513Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=grafanaStorageLogger t=2024-03-11T22:00:13.60873555Z level=info msg="storage starting" 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=http.server t=2024-03-11T22:00:13.610860341Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T22:00:13.850 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:00:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a[38359]: logger=sqlstore.transactions t=2024-03-11T22:00:13.628970174Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:00:14.217 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:14 smithi189 ceph-mon[27727]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:14.253 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:14 smithi176 ceph-mon[27622]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:14.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:14 smithi151 ceph-mon[25773]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:14.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T22:00:15.053 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:15.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:15 smithi151 ceph-mon[25773]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:15.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:15 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:15.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:15 smithi176 ceph-mon[27622]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:15.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:15 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:15.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:15 smithi189 ceph-mon[27727]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:15.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:15 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' 2024-03-11T22:00:16.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:16 smithi176 ceph-mon[27622]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:16.377 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:16 smithi189 ceph-mon[27727]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:16.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:16 smithi151 ceph-mon[25773]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: agent 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: agent 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout: host_pattern: '*' 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:59:34.541526Z' 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout: last_refresh: '2024-03-11T21:59:07.974495Z' 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout: running: 3 2024-03-11T22:00:17.198 INFO:teuthology.orchestra.run.smithi151.stdout: size: 3 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:events: 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:- 2024-03-11T21:59:35.941265Z service:agent [INFO] "service was created" 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: alertmanager 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: alertmanager 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: count: 1 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi151=a 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:59:04.020050Z' 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: ports: 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: - 9093 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: - 9094 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: running: 0 2024-03-11T22:00:17.199 INFO:teuthology.orchestra.run.smithi151.stdout: size: 1 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:events: 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:- 2024-03-11T21:59:50.899928Z service:alertmanager [INFO] "service was created" 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: grafana 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: grafana 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: count: 1 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi176=a 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:spec: 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: anonymous_access: true 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: protocol: https 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:59:06.619368Z' 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: ports: 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: - 3000 2024-03-11T22:00:17.200 INFO:teuthology.orchestra.run.smithi151.stdout: running: 0 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: size: 1 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:events: 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:- 2024-03-11T22:00:12.125313Z service:grafana [INFO] "service was created" 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: mgr 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: mgr 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: count: 3 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi151=y 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi176=x 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi189=z 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:57:19.632680Z' 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: last_refresh: '2024-03-11T21:59:07.974665Z' 2024-03-11T22:00:17.201 INFO:teuthology.orchestra.run.smithi151.stdout: running: 3 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: size: 3 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: mon 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: mon 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: count: 3 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi151:172.21.15.151=a 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi176:172.21.15.176=b 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi189:172.21.15.189=c 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.202 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:56:53.117919Z' 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: last_refresh: '2024-03-11T21:59:07.974591Z' 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: running: 3 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: size: 3 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: node-exporter 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: node-exporter 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: count: 3 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi151=a 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi176=b 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi189=c 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:59:01.614241Z' 2024-03-11T22:00:17.203 INFO:teuthology.orchestra.run.smithi151.stdout: ports: 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: - 9100 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: running: 0 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: size: 3 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: osd 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: osd 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:unmanaged: true 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:spec: 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: filter_logic: AND 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: objectstore: bluestore 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: container_image_id: a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: last_refresh: '2024-03-11T21:59:07.974737Z' 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: running: 3 2024-03-11T22:00:17.204 INFO:teuthology.orchestra.run.smithi151.stdout: size: 3 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: prometheus 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: prometheus 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: count: 1 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi176=a 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:58:58.988594Z' 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: ports: 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: - 9095 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: running: 0 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout: size: 1 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:--- 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:service_type: rgw 2024-03-11T22:00:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:service_id: foo.a 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout:service_name: rgw.foo.a 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout:placement: 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: count: 1 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: hosts: 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: - smithi151=foo.a 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout:status: 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: created: '2024-03-11T21:58:59.535813Z' 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: last_refresh: '2024-03-11T21:59:23.821209Z' 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: ports: 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: - 80 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: running: 1 2024-03-11T22:00:17.206 INFO:teuthology.orchestra.run.smithi151.stdout: size: 1 2024-03-11T22:00:18.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:18 smithi151 ceph-mon[25773]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:18.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:18 smithi151 ceph-mon[25773]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:00:18.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:18 smithi176 ceph-mon[27622]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:18.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:18 smithi176 ceph-mon[27622]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:00:18.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:18 smithi189 ceph-mon[27727]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:18.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:18 smithi189 ceph-mon[27727]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:00:18.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T22:00:19.066 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:00:20.371 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:20 smithi151 ceph-mon[25773]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:20.371 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:20 smithi151 ceph-mon[25773]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:00:20.371 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:20 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ::ffff:172.21.15.176 - - [11/Mar/2024:22:00:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:00:20.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:20 smithi176 ceph-mon[27622]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:20.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:20 smithi176 ceph-mon[27622]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:00:20.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:20 smithi189 ceph-mon[27727]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:20.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:20 smithi189 ceph-mon[27727]: from='mgr.24328 172.21.15.151:0/421414374' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:00:20.742 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:20 smithi151 bash[51568]: Writing manifest to image destination 2024-03-11T22:00:20.742 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:20 smithi151 podman[51568]: 2024-03-11 22:00:20.708861394 +0000 UTC m=+4.728176858 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:00:21.718 INFO:teuthology.orchestra.run.smithi151.stdout:osd 3 73s ago - 2024-03-11T22:00:22.430 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:22 smithi189 ceph-mon[27727]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:22.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:22 smithi151 ceph-mon[25773]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:22.599 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:22 smithi176 ceph-mon[27622]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:00:22.626 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T22:00:22.635 INFO:tasks.cephadm:Teardown begin 2024-03-11T22:00:22.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:00:22.667 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:00:22.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:00:22.740 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T22:00:22.740 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:00:22.756 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:00:22.772 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:00:22.796 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T22:00:22.796 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T22:00:22.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:00:22.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:22 smithi151 systemd[1]: Stopping Ceph mon.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:22.898 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:22 smithi151 podman[51568]: 2024-03-11 22:00:22.760519382 +0000 UTC m=+6.779834845 container create 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:00:23.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-mon[25773]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:23.304 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 podman[51568]: 2024-03-11 22:00:23.004772329 +0000 UTC m=+7.024087792 container init 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:00:23.304 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 podman[51568]: 2024-03-11 22:00:23.010165457 +0000 UTC m=+7.029480925 container start 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:00:23.304 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.014Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:00:23.304 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.014Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.014Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.014Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.014Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:00:23.305 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:00:23.306 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:00:23.306 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:00:23.306 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:00:23.306 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:00:23.306 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:00:23.306 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:00:23.307 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:00:23.308 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a[52339]: ts=2024-03-11T22:00:23.015Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 bash[51568]: 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d 2024-03-11T22:00:23.309 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:00:23 smithi151 systemd[1]: Started Ceph node-exporter.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:00:23.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:23 smithi176 ceph-mon[27622]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a[25750]: 2024-03-11T22:00:23.309+0000 7fd8a8138700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a[25750]: 2024-03-11T22:00:23.309+0000 7fd8a8138700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T22:00:23.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:23 smithi151 podman[52385]: 2024-03-11 22:00:23.540125535 +0000 UTC m=+0.357680170 container died d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T22:00:23.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:23] ENGINE Bus STOPPING 2024-03-11T22:00:23.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:00:23.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:23] ENGINE Bus STOPPED 2024-03-11T22:00:23.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:23] ENGINE Bus STARTING 2024-03-11T22:00:23.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:23] ENGINE Serving on http://:::9283 2024-03-11T22:00:23.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:23 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:23] ENGINE Bus STARTED 2024-03-11T22:00:23.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:23 smithi189 ceph-mon[27727]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:00:24.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:23 smithi151 podman[52385]: 2024-03-11 22:00:23.958426878 +0000 UTC m=+0.775981518 container cleanup d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T22:00:24.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:23 smithi151 bash[52385]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:00:24.762 INFO:journalctl@ceph.mon.a.smithi151.stdout:Mar 11 22:00:24 smithi151 podman[52443]: 2024-03-11 22:00:24.595103982 +0000 UTC m=+1.233532411 container remove d72ded8e46bc14b54223d91bbcd0b2b648fa251bc8a7693221935420d9acda64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T22:00:24.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T22:00:24.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:24.824 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T22:00:24.824 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T22:00:24.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.b 2024-03-11T22:00:25.324 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:25 smithi176 systemd[1]: Stopping Ceph mon.b for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:25.324 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:25 smithi176 systemd[1]: Starting Ceph node-exporter.b for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:25.599 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:25 smithi176 bash[39449]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:00:26.415 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b[27599]: 2024-03-11T22:00:26.267+0000 7f4c0abd7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:26.415 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:26 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b[27599]: 2024-03-11T22:00:26.267+0000 7f4c0abd7700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:00:26.415 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:26 smithi176 bash[39449]: Getting image source signatures 2024-03-11T22:00:26.416 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:26 smithi176 bash[39449]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:00:26.416 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:26 smithi176 bash[39449]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:00:26.416 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:26 smithi176 bash[39449]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:00:26.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:26] ENGINE Bus STOPPING 2024-03-11T22:00:26.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:00:26.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:26] ENGINE Bus STOPPED 2024-03-11T22:00:26.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:26] ENGINE Bus STARTING 2024-03-11T22:00:26.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:26] ENGINE Serving on http://:::9283 2024-03-11T22:00:26.556 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:26 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: [11/Mar/2024:22:00:26] ENGINE Bus STARTED 2024-03-11T22:00:26.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:26 smithi176 podman[39475]: 2024-03-11 22:00:26.421533896 +0000 UTC m=+1.144386655 container died 3fcc03fa32282a53bacb2c44cff19ae1ed9e2d2d849019eedda50c2ee2967cc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T22:00:30.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:30 smithi176 podman[39475]: 2024-03-11 22:00:30.178796712 +0000 UTC m=+4.901649475 container cleanup 3fcc03fa32282a53bacb2c44cff19ae1ed9e2d2d849019eedda50c2ee2967cc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T22:00:30.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:30 smithi176 bash[39475]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b 2024-03-11T22:00:30.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:30 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: ::ffff:172.21.15.176 - - [11/Mar/2024:22:00:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:00:30.658 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 11 22:00:30 smithi176 podman[39494]: 2024-03-11 22:00:30.514084201 +0000 UTC m=+4.085315925 container remove 3fcc03fa32282a53bacb2c44cff19ae1ed9e2d2d849019eedda50c2ee2967cc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T22:00:31.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T22:00:32.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:32.029 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T22:00:32.029 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T22:00:32.029 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.c 2024-03-11T22:00:32.322 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:32 smithi189 systemd[1]: Stopping Ceph mon.c for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:32.349 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 bash[39449]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:00:32.627 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:32 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-c[27691]: 2024-03-11T22:00:32.327+0000 7f56503b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:32.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:32 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-c[27691]: 2024-03-11T22:00:32.327+0000 7f56503b5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:00:32.628 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:32 smithi189 podman[39169]: 2024-03-11 22:00:32.568744791 +0000 UTC m=+0.334049640 container died 69d0055a18d1f9cc66a075c07c2b1063ca8834bfb271a13e922435190e5b7fb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:00:32.761 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 bash[39449]: Writing manifest to image destination 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 podman[39449]: 2024-03-11 22:00:32.694490997 +0000 UTC m=+7.468323393 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 podman[39449]: 2024-03-11 22:00:32.776384779 +0000 UTC m=+7.550217175 container create 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 podman[39449]: 2024-03-11 22:00:32.986816446 +0000 UTC m=+7.760648842 container init 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 podman[39449]: 2024-03-11 22:00:32.992246738 +0000 UTC m=+7.766079135 container start 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.996Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.996Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.996Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.996Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:00:33.056 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.996Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.996Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:00:33.057 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:00:33.058 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:00:33.059 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:32 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b[39659]: ts=2024-03-11T22:00:32.997Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:00:33.060 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:33 smithi176 bash[39449]: 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 2024-03-11T22:00:33.074 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:32 smithi189 podman[39169]: 2024-03-11 22:00:32.902278378 +0000 UTC m=+0.667583234 container cleanup 69d0055a18d1f9cc66a075c07c2b1063ca8834bfb271a13e922435190e5b7fb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-11T22:00:33.074 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:32 smithi189 bash[39169]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-c 2024-03-11T22:00:33.349 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:00:33 smithi176 systemd[1]: Started Ceph node-exporter.b for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:00:34.121 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 11 22:00:33 smithi189 podman[39202]: 2024-03-11 22:00:33.799490544 +0000 UTC m=+1.405911217 container remove 69d0055a18d1f9cc66a075c07c2b1063ca8834bfb271a13e922435190e5b7fb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T22:00:34.469 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T22:00:34.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:34.523 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T22:00:34.523 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T22:00:34.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:00:34.854 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:34 smithi151 systemd[1]: Stopping Ceph mgr.y for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:35.241 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:34 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y[26025]: 2024-03-11T22:00:34.858+0000 7f929ee64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:35.242 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:34 smithi151 podman[52813]: 2024-03-11 22:00:34.890203846 +0000 UTC m=+0.159091859 container died 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T22:00:35.242 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:35 smithi151 podman[52813]: 2024-03-11 22:00:35.248190664 +0000 UTC m=+0.517078677 container cleanup 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:00:35.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:35 smithi151 bash[52813]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y 2024-03-11T22:00:35.555 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Mar 11 22:00:35 smithi151 podman[52832]: 2024-03-11 22:00:35.416125239 +0000 UTC m=+0.526305379 container remove 6348834d01f90de76641bad66750899a0cb6733440dac2af58e86655b3d7b041 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T22:00:35.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T22:00:35.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:35.699 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T22:00:35.699 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T22:00:35.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.x 2024-03-11T22:00:36.004 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 22:00:35 smithi176 systemd[1]: Stopping Ceph mgr.x for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:36.282 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 22:00:36 smithi176 podman[39855]: 2024-03-11 22:00:36.01074472 +0000 UTC m=+0.110407269 container died c86fd27ac212532b555187886f5c8b22d4f7afe82b242199ecdfdf2e6a04d2e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T22:00:36.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 22:00:36 smithi176 podman[39855]: 2024-03-11 22:00:36.376764358 +0000 UTC m=+0.476426906 container cleanup c86fd27ac212532b555187886f5c8b22d4f7afe82b242199ecdfdf2e6a04d2e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T22:00:36.599 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 11 22:00:36 smithi176 bash[39855]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-x 2024-03-11T22:00:36.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T22:00:36.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:36.838 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T22:00:36.838 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T22:00:36.839 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.z 2024-03-11T22:00:37.127 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 22:00:36 smithi189 systemd[1]: Stopping Ceph mgr.z for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:37.493 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 22:00:37 smithi189 podman[39479]: 2024-03-11 22:00:37.190416678 +0000 UTC m=+0.148767665 container died 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T22:00:37.493 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 22:00:37 smithi189 podman[39479]: 2024-03-11 22:00:37.500126929 +0000 UTC m=+0.458477916 container cleanup 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-11T22:00:37.771 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 22:00:37 smithi189 bash[39479]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z 2024-03-11T22:00:37.771 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 22:00:37 smithi189 podman[39498]: 2024-03-11 22:00:37.670070627 +0000 UTC m=+0.474876323 container remove 79be1823948d3587e6e5c1bcef171d900d58544e3c8926cfdbce1fb58ed002f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T22:00:37.771 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 11 22:00:37 smithi189 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:00:37.789 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T22:00:37.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:37.826 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T22:00:37.826 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T22:00:37.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.0 2024-03-11T22:00:38.123 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:37 smithi151 systemd[1]: Stopping Ceph osd.0 for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:38.389 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:38 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0[38152]: 2024-03-11T22:00:38.128+0000 7f67dd4b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:38.389 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:38 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0[38152]: 2024-03-11T22:00:38.128+0000 7f67dd4b4700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-11T22:00:38.389 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:38 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0[38152]: 2024-03-11T22:00:38.129+0000 7f67dd4b4700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:00:43.964 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:43 smithi151 podman[52942]: 2024-03-11 22:00:43.644317401 +0000 UTC m=+5.621022767 container died 66c24425769c7a26a9b9f08dfed1a9552895d754b704a0289ab3339bf4228cfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:00:44.265 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[52942]: 2024-03-11 22:00:44.119058584 +0000 UTC m=+6.095763989 container cleanup 66c24425769c7a26a9b9f08dfed1a9552895d754b704a0289ab3339bf4228cfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T22:00:44.266 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 bash[52942]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:00:44.555 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[53448]: 2024-03-11 22:00:44.270169015 +0000 UTC m=+0.619406745 container remove 66c24425769c7a26a9b9f08dfed1a9552895d754b704a0289ab3339bf4228cfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T22:00:44.908 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[53545]: 2024-03-11 22:00:44.662326221 +0000 UTC m=+0.106999495 container create c7d1faf4907d01ee17f2fd36862b153a9bdcc726988e3961dcc714a3861eb1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T22:00:44.908 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[53545]: 2024-03-11 22:00:44.572889653 +0000 UTC m=+0.017562937 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:00:44.908 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[53545]: 2024-03-11 22:00:44.831711765 +0000 UTC m=+0.276385050 container init c7d1faf4907d01ee17f2fd36862b153a9bdcc726988e3961dcc714a3861eb1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:00:44.908 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[53545]: 2024-03-11 22:00:44.838147296 +0000 UTC m=+0.282820572 container start c7d1faf4907d01ee17f2fd36862b153a9bdcc726988e3961dcc714a3861eb1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T22:00:44.908 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:44 smithi151 podman[53545]: 2024-03-11 22:00:44.914275024 +0000 UTC m=+0.358948320 container attach c7d1faf4907d01ee17f2fd36862b153a9bdcc726988e3961dcc714a3861eb1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:00:45.305 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:45 smithi151 podman[53545]: 2024-03-11 22:00:45.124293213 +0000 UTC m=+0.568966488 container died c7d1faf4907d01ee17f2fd36862b153a9bdcc726988e3961dcc714a3861eb1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:00:45.663 INFO:journalctl@ceph.osd.0.smithi151.stdout:Mar 11 22:00:45 smithi151 podman[53592]: 2024-03-11 22:00:45.650599846 +0000 UTC m=+0.517423620 container remove c7d1faf4907d01ee17f2fd36862b153a9bdcc726988e3961dcc714a3861eb1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:00:45.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T22:00:45.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:45.708 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T22:00:45.708 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T22:00:45.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.1 2024-03-11T22:00:46.013 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:45 smithi176 systemd[1]: Stopping Ceph osd.1 for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:46.808 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:46 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1[33297]: 2024-03-11T22:00:46.475+0000 7fcbb30f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:46.808 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:46 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1[33297]: 2024-03-11T22:00:46.475+0000 7fcbb30f1700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-11T22:00:46.808 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:46 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1[33297]: 2024-03-11T22:00:46.475+0000 7fcbb30f1700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:00:52.134 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:51 smithi176 podman[40603]: 2024-03-11 22:00:51.881594416 +0000 UTC m=+5.534604872 container died c9106bbde9c981cf0864c89c2e51e8fdbb2546527d51f1952df7667a62a25563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:00:52.508 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:52 smithi176 podman[40603]: 2024-03-11 22:00:52.247169011 +0000 UTC m=+5.900179469 container cleanup c9106bbde9c981cf0864c89c2e51e8fdbb2546527d51f1952df7667a62a25563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:00:52.508 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:52 smithi176 bash[40603]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:00:52.849 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:52 smithi176 podman[40755]: 2024-03-11 22:00:52.514650776 +0000 UTC m=+0.633652338 container remove c9106bbde9c981cf0864c89c2e51e8fdbb2546527d51f1952df7667a62a25563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:00:53.161 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:52 smithi176 podman[40850]: 2024-03-11 22:00:52.76571965 +0000 UTC m=+0.017954987 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:00:53.161 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:52 smithi176 podman[40850]: 2024-03-11 22:00:52.865371525 +0000 UTC m=+0.117606891 container create 6cfee54a1a09ac738dddd0aab44a8f4643e7aa419ae096723006df45a6f520a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-11T22:00:53.162 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:53 smithi176 podman[40850]: 2024-03-11 22:00:53.084454049 +0000 UTC m=+0.336689376 container init 6cfee54a1a09ac738dddd0aab44a8f4643e7aa419ae096723006df45a6f520a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:00:53.162 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:53 smithi176 podman[40850]: 2024-03-11 22:00:53.08971623 +0000 UTC m=+0.341951559 container start 6cfee54a1a09ac738dddd0aab44a8f4643e7aa419ae096723006df45a6f520a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T22:00:53.162 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:53 smithi176 podman[40850]: 2024-03-11 22:00:53.167809226 +0000 UTC m=+0.420044551 container attach 6cfee54a1a09ac738dddd0aab44a8f4643e7aa419ae096723006df45a6f520a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-11T22:00:53.599 INFO:journalctl@ceph.osd.1.smithi176.stdout:Mar 11 22:00:53 smithi176 podman[40850]: 2024-03-11 22:00:53.378206102 +0000 UTC m=+0.630441437 container died 6cfee54a1a09ac738dddd0aab44a8f4643e7aa419ae096723006df45a6f520a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T22:00:53.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T22:00:54.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:00:54.019 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T22:00:54.019 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T22:00:54.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.2 2024-03-11T22:00:54.277 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:00:54 smithi189 systemd[1]: Stopping Ceph osd.2 for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:00:55.283 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:00:54 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2[34411]: 2024-03-11T22:00:54.908+0000 7f89b9ae2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:00:55.283 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:00:54 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2[34411]: 2024-03-11T22:00:54.908+0000 7f89b9ae2700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-11T22:00:55.283 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:00:54 smithi189 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2[34411]: 2024-03-11T22:00:54.908+0000 7f89b9ae2700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:01:00.628 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:00 smithi189 podman[40015]: 2024-03-11 22:01:00.367669588 +0000 UTC m=+5.568031296 container died 1ca0bef246f3fbe10fd261db04f9d1bbe8d5372126b5c3d46f1bde864419d04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T22:01:00.969 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:00 smithi189 podman[40015]: 2024-03-11 22:01:00.733664255 +0000 UTC m=+5.934025971 container cleanup 1ca0bef246f3fbe10fd261db04f9d1bbe8d5372126b5c3d46f1bde864419d04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:01:00.970 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:00 smithi189 bash[40015]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:01:01.295 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:00 smithi189 podman[40118]: 2024-03-11 22:01:00.975993143 +0000 UTC m=+0.602851379 container remove 1ca0bef246f3fbe10fd261db04f9d1bbe8d5372126b5c3d46f1bde864419d04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T22:01:01.572 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:01 smithi189 podman[40212]: 2024-03-11 22:01:01.301745974 +0000 UTC m=+0.110881334 container create a740a2dc49875a4aa8b5b2dbdb2ad515e7b74fddcddc9e7c408c6b7b9438b131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T22:01:01.573 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:01 smithi189 podman[40212]: 2024-03-11 22:01:01.209778469 +0000 UTC m=+0.018913847 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:01:01.573 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:01 smithi189 podman[40212]: 2024-03-11 22:01:01.499266475 +0000 UTC m=+0.308401836 container init a740a2dc49875a4aa8b5b2dbdb2ad515e7b74fddcddc9e7c408c6b7b9438b131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T22:01:01.573 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:01 smithi189 podman[40212]: 2024-03-11 22:01:01.504868096 +0000 UTC m=+0.314003459 container start a740a2dc49875a4aa8b5b2dbdb2ad515e7b74fddcddc9e7c408c6b7b9438b131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:01:01.573 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:01 smithi189 podman[40212]: 2024-03-11 22:01:01.578846092 +0000 UTC m=+0.387981455 container attach a740a2dc49875a4aa8b5b2dbdb2ad515e7b74fddcddc9e7c408c6b7b9438b131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:01:01.849 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 11 22:01:01 smithi189 podman[40212]: 2024-03-11 22:01:01.803948046 +0000 UTC m=+0.613083416 container died a740a2dc49875a4aa8b5b2dbdb2ad515e7b74fddcddc9e7c408c6b7b9438b131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:01:02.291 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T22:01:02.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:01:02.332 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T22:01:02.332 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T22:01:02.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T22:01:02.652 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:02 smithi151 systemd[1]: Stopping Ceph rgw.foo.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:01:03.055 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:02 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a[41454]: 2024-03-11T22:01:02.657+0000 7f6638ae6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:01:03.055 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:02 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a[41454]: 2024-03-11T22:01:02.657+0000 7f6647356ac0 -1 shutting down 2024-03-11T22:01:12.945 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:12 smithi151 bash[54367]: time="2024-03-11T22:01:12Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T22:01:12.945 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:12 smithi151 podman[54367]: 2024-03-11 22:01:12.680640763 +0000 UTC m=+10.145082682 container died b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T22:01:13.286 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 podman[54367]: 2024-03-11 22:01:13.099963519 +0000 UTC m=+10.564405447 container cleanup b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:01:13.286 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 bash[54367]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T22:01:13.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T22:01:13.552 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 podman[54567]: 2024-03-11 22:01:13.292614818 +0000 UTC m=+0.607425730 container remove b2d47ebb63d9707ad3453b268131a9b5ae32e9dba3f9f019e3e79af33b3758eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T22:01:13.552 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T22:01:13.552 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-11T22:01:13.552 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 systemd[1]: Stopped Ceph rgw.foo.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:01:13.552 INFO:journalctl@ceph.rgw.foo.a.smithi151.stdout:Mar 11 22:01:13 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@rgw.foo.a.service: Consumed 1.656s CPU time. 2024-03-11T22:01:13.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:01:13.567 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T22:01:13.567 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T22:01:13.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T22:01:13.875 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 systemd[1]: Stopping Ceph prometheus.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:01:14.156 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a[36099]: ts=2024-03-11T22:01:13.881Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:01:14.157 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:13 smithi176 podman[41706]: 2024-03-11 22:01:13.892334547 +0000 UTC m=+0.119232186 container died b079a4546be4adb16b5b0e100beb628db64800abb03240a4128cafc210e5800f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:14.531 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:14 smithi176 podman[41706]: 2024-03-11 22:01:14.261947794 +0000 UTC m=+0.488845425 container cleanup b079a4546be4adb16b5b0e100beb628db64800abb03240a4128cafc210e5800f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:14.531 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 11 22:01:14 smithi176 bash[41706]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:01:14.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T22:01:14.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:01:14.712 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T22:01:14.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:01:14.909 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T22:01:16.055 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:15 smithi151 systemd[1]: Stopping Ceph alertmanager.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:01:16.555 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:16 smithi151 ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a[48891]: ts=2024-03-11T22:01:16.307Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:01:16.555 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:16 smithi151 podman[55220]: 2024-03-11 22:01:16.328047224 +0000 UTC m=+0.142856401 container died 8842803b826069be5c231cb79ae75caaab8c873f6a679a5f60aae1782558841a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:17.041 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:16 smithi151 podman[55220]: 2024-03-11 22:01:16.779769991 +0000 UTC m=+0.594579161 container cleanup 8842803b826069be5c231cb79ae75caaab8c873f6a679a5f60aae1782558841a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:17.041 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:16 smithi151 bash[55220]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:01:17.305 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:17 smithi151 podman[55239]: 2024-03-11 22:01:17.047738964 +0000 UTC m=+0.717571660 container remove 8842803b826069be5c231cb79ae75caaab8c873f6a679a5f60aae1782558841a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:17.305 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:17 smithi151 podman[55239]: 2024-03-11 22:01:17.123326543 +0000 UTC m=+0.793159238 volume remove 19997ada5c587e337d2820fad322188a0a924f6e96eead3f77885d3dd21f23d8 2024-03-11T22:01:17.305 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:17 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:01:17.305 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:17 smithi151 systemd[1]: Stopped Ceph alertmanager.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:01:17.305 INFO:journalctl@ceph.alertmanager.a.smithi151.stdout:Mar 11 22:01:17 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.054s CPU time. 2024-03-11T22:01:17.699 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:17 smithi151 systemd[1]: Stopping Ceph node-exporter.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:01:18.052 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:17 smithi151 podman[55358]: 2024-03-11 22:01:17.706158286 +0000 UTC m=+0.119697954 container died 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:18.053 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 podman[55358]: 2024-03-11 22:01:18.058966544 +0000 UTC m=+0.472506230 container cleanup 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:18.303 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 bash[55358]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T22:01:18.555 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 podman[55377]: 2024-03-11 22:01:18.31001692 +0000 UTC m=+0.605195022 container remove 02ce6ef4ea5f25541a78d3b5660bfe10e62bf1b4a130ba6629563e5e9563a21d (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:01:18.555 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:01:18.555 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T22:01:18.555 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 systemd[1]: Stopped Ceph node-exporter.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:01:18.555 INFO:journalctl@ceph.node-exporter.a.smithi151.stdout:Mar 11 22:01:18 smithi151 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.912s CPU time. 2024-03-11T22:01:19.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:01:19.351 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T22:01:20.599 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:20 smithi176 systemd[1]: Stopping Ceph grafana.a for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:01:20.923 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:20 smithi176 podman[42513]: 2024-03-11 22:01:20.648689817 +0000 UTC m=+0.125831613 container died 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a, com.redhat.component=ubi8-container, name=ubi8, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, release=239.1651231664, io.buildah.version=1.32.0, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-11T22:01:20.923 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:20 smithi176 podman[42513]: 2024-03-11 22:01:20.929467287 +0000 UTC m=+0.406609077 container cleanup 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vendor=Red Hat, Inc., version=8.5, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, description=Ceph Grafana Container, architecture=x86_64, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-11T22:01:21.262 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:20 smithi176 bash[42513]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a 2024-03-11T22:01:21.263 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:21 smithi176 podman[42532]: 2024-03-11 22:01:21.155234891 +0000 UTC m=+0.509102713 container remove 18f398d447623c4b719c258ec308a6639b08752b994182ae4557627b83205cf6 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, version=8.5, name=ubi8, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.buildah.version=1.32.0, release=239.1651231664, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner ) 2024-03-11T22:01:21.263 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:21 smithi176 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:01:21.599 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:21 smithi176 systemd[1]: Stopping Ceph node-exporter.b for e42c6b1c-dff1-11ee-95c8-87774f69a715... 2024-03-11T22:01:21.599 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:21 smithi176 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T22:01:21.599 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:21 smithi176 systemd[1]: Stopped Ceph grafana.a for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:01:21.600 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 11 22:01:21 smithi176 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@grafana.a.service: Consumed 3.693s CPU time. 2024-03-11T22:01:22.019 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:21 smithi176 podman[42650]: 2024-03-11 22:01:21.746146718 +0000 UTC m=+0.111573161 container died 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:01:22.349 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 podman[42650]: 2024-03-11 22:01:22.124524365 +0000 UTC m=+0.489950809 container cleanup 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:01:22.349 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 bash[42650]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T22:01:22.627 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 podman[42669]: 2024-03-11 22:01:22.358908083 +0000 UTC m=+0.617676436 container remove 15026148fa1fe335520093ff3fe0fdf40b498c1efa83fbc89c2a3fbd72a4d104 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:01:22.627 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:01:22.627 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T22:01:22.627 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 systemd[1]: Stopped Ceph node-exporter.b for e42c6b1c-dff1-11ee-95c8-87774f69a715. 2024-03-11T22:01:22.627 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 11 22:01:22 smithi176 systemd[1]: ceph-e42c6b1c-dff1-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 1.859s CPU time. 2024-03-11T22:01:23.115 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:01:23.312 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T22:01:24.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:01:24.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:01:24.781 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:01:24.811 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T22:01:24.812 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi151/crash 2024-03-11T22:01:24.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:01:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:01:24.857 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:01:24.859 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi176/crash 2024-03-11T22:01:24.860 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:01:24.900 INFO:teuthology.orchestra.run.smithi176.stderr:tar: /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:01:24.900 INFO:teuthology.orchestra.run.smithi176.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:01:24.902 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi189/crash 2024-03-11T22:01:24.903 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:01:24.938 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:01:24.938 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:01:24.939 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T22:01:24.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:01:24.970 INFO:teuthology.orchestra.run.smithi151.stdout:2024-03-11T21:57:11.739599+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:01:24.971 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T22:01:24.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[SEC\]' /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:01:25.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]' /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:01:25.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[WRN\]' /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:01:25.180 INFO:teuthology.orchestra.run.smithi151.stdout:2024-03-11T21:57:11.739599+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:01:25.181 INFO:tasks.cephadm:Compressing logs... 2024-03-11T22:01:25.181 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:01:25.224 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:01:25.226 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:01:25.249 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:01:25.249 INFO:teuthology.orchestra.run.smithi151.stderr:: No such file or directory 2024-03-11T22:01:25.249 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T22:01:25.249 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:01:25.250 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T22:01:25.251 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T22:01:25.251 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T22:01:25.251 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.y.log: 57.7% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 89.8% --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr: -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log: 82.6% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log: 84.7% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:01:25.253 INFO:teuthology.orchestra.run.smithi151.stderr: 86.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:01:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-11T22:01:25.254 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-11T22:01:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:01:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T22:01:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:01:25.256 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:01:25.256 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T22:01:25.256 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:01:25.256 INFO:teuthology.orchestra.run.smithi189.stderr: 83.9% -- replaced with /var/log/ceph/cephadm.log.gz 81.0% 2024-03-11T22:01:25.257 INFO:teuthology.orchestra.run.smithi189.stderr: -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:01:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T22:01:25.257 INFO:teuthology.orchestra.run.smithi189.stderr: 89.9% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:01:25.257 INFO:teuthology.orchestra.run.smithi189.stderr: 84.3% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:: No such file or directory 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:01:25.258 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:01:25.260 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.x.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:01:25.260 INFO:teuthology.orchestra.run.smithi176.stderr: 89.8% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:01:25.260 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T22:01:25.260 INFO:teuthology.orchestra.run.smithi176.stderr: 80.9% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:01:25.260 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.b.log: 79.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:01:25.261 INFO:teuthology.orchestra.run.smithi176.stderr: 84.2% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:01:25.261 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T22:01:25.264 INFO:teuthology.orchestra.run.smithi176.stderr: 90.2% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T22:01:25.267 INFO:teuthology.orchestra.run.smithi151.stderr: 90.1% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:01:25.270 INFO:teuthology.orchestra.run.smithi189.stderr: 90.2% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:01:25.274 INFO:teuthology.orchestra.run.smithi176.stderr: 90.2% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:01:25.316 INFO:teuthology.orchestra.run.smithi189.stderr: 92.3% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T22:01:25.316 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T22:01:25.327 INFO:teuthology.orchestra.run.smithi151.stderr: 88.4% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T22:01:25.450 INFO:teuthology.orchestra.run.smithi151.stderr: 91.5% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T22:01:25.518 INFO:teuthology.orchestra.run.smithi151.stderr: 94.3% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:01:25.518 INFO:teuthology.orchestra.run.smithi189.stderr: 94.3% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T22:01:25.519 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-11T22:01:25.519 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m0.284s 2024-03-11T22:01:25.519 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m0.558s 2024-03-11T22:01:25.520 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.040s 2024-03-11T22:01:25.520 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-11T22:01:25.520 INFO:teuthology.orchestra.run.smithi189.stderr:real 0m0.280s 2024-03-11T22:01:25.521 INFO:teuthology.orchestra.run.smithi189.stderr:user 0m0.347s 2024-03-11T22:01:25.521 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m0.034s 2024-03-11T22:01:25.569 INFO:teuthology.orchestra.run.smithi176.stderr: 94.3% -- replaced with /var/log/ceph/e42c6b1c-dff1-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:01:25.571 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-11T22:01:25.572 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m0.331s 2024-03-11T22:01:25.572 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m0.390s 2024-03-11T22:01:25.572 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.045s 2024-03-11T22:01:25.572 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:01:25.572 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi151/log 2024-03-11T22:01:25.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:01:25.696 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi176/log 2024-03-11T22:01:25.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:01:25.787 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi189/log 2024-03-11T22:01:25.788 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:01:25.869 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:01:25.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --force 2024-03-11T22:01:26.051 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T22:01:26.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --force 2024-03-11T22:01:26.348 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T22:01:26.463 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e42c6b1c-dff1-11ee-95c8-87774f69a715 --force 2024-03-11T22:01:26.646 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: e42c6b1c-dff1-11ee-95c8-87774f69a715 2024-03-11T22:01:26.764 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:01:26.764 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:26.780 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:26.796 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:26.810 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:01:26.811 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T22:01:26.820 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_1... 2024-03-11T22:01:26.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_1 2024-03-11T22:01:26.959 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:01:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:26.961 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_2... 2024-03-11T22:01:26.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_2 2024-03-11T22:01:27.095 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:01:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:27.098 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_3... 2024-03-11T22:01:27.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_3 2024-03-11T22:01:27.284 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:01:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:27.286 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_4... 2024-03-11T22:01:27.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_4 2024-03-11T22:01:27.464 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:01:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:27.466 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-11T22:01:27.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/scratch_devs 2024-03-11T22:01:27.493 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_1... 2024-03-11T22:01:27.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_1 2024-03-11T22:01:27.642 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:01:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:27.644 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_2... 2024-03-11T22:01:27.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_2 2024-03-11T22:01:27.791 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:01:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:27.793 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_3... 2024-03-11T22:01:27.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_3 2024-03-11T22:01:27.930 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:01:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:27.932 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_4... 2024-03-11T22:01:27.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_4 2024-03-11T22:01:28.075 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:01:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:28.077 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-11T22:01:28.077 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/scratch_devs 2024-03-11T22:01:28.107 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_1... 2024-03-11T22:01:28.107 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_1 2024-03-11T22:01:28.254 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:01:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:28.256 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_2... 2024-03-11T22:01:28.256 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_2 2024-03-11T22:01:28.399 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:01:28.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:28.401 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_3... 2024-03-11T22:01:28.401 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_3 2024-03-11T22:01:28.532 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:01:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:28.534 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_4... 2024-03-11T22:01:28.534 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_4 2024-03-11T22:01:28.679 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:01:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:28.681 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-11T22:01:28.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/scratch_devs 2024-03-11T22:01:28.710 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:01:28.719 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:01:28.719 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:01:28.721 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:01:28.724 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:01:28.734 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:01:28.738 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:01:28.738 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-11T22:01:28.738 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +4850us[+4850us] +/- 106ms 2024-03-11T22:01:28.738 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -2097us[-2095us] +/- 54ms 2024-03-11T22:01:28.739 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -591us[ -588us] +/- 82ms 2024-03-11T22:01:28.739 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:01:28.739 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:01:28.743 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:01:28.743 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-11T22:01:28.743 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +4853us[+4853us] +/- 106ms 2024-03-11T22:01:28.743 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -2096us[-2096us] +/- 54ms 2024-03-11T22:01:28.743 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -585us[ -584us] +/- 83ms 2024-03-11T22:01:28.743 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:01:28.765 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:01:28.769 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:01:28.770 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-11T22:01:28.770 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +4849us[+4849us] +/- 106ms 2024-03-11T22:01:28.770 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 57 -2099us[-2079us] +/- 54ms 2024-03-11T22:01:28.770 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv03.front.sepia.ceph.com 3 6 377 57 -586us[ -566us] +/- 83ms 2024-03-11T22:01:28.770 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:01:28.771 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:01:28.780 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:01:28.781 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:01:28.789 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:01:28.863 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:01:28.937 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:01:29.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:01:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:29.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:01:29.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:29.075 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:01:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:29.106 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:01:29.118 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:01:29.127 INFO:teuthology.task.internal:Duration was 704.940044 seconds 2024-03-11T22:01:29.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:01:29.142 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:01:29.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:01:29.145 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:01:29.148 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:01:29.187 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:01:29.190 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:01:29.194 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:01:29.637 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:01:29.637 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-03-11T22:01:29.638 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:01:29.657 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-03-11T22:01:29.658 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:01:29.678 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-03-11T22:01:29.679 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:01:29.710 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:01:29.710 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:01:29.713 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:01:29.721 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:01:29.779 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:01:29.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:01:29.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:01:29.789 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:01:29.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:01:29.988 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:01:29.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:01:30.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:01:30.042 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:01:30.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:01:30.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:01:30.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:01:30.085 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:01:30.104 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-03-11T22:01:30.107 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-03-11T22:01:30.135 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-03-11T22:01:30.157 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:01:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:30.185 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:01:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:30.199 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:01:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:01:30.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:01:30.224 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:01:30.225 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi151 2024-03-11T22:01:30.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:01:30.264 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi176 2024-03-11T22:01:30.264 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:01:30.301 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947/remote/smithi189 2024-03-11T22:01:30.301 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:01:30.337 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:01:30.338 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:01:30.340 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:01:30.342 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:01:30.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:01:30.398 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:01:30.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:01:30.408 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:01:30.408 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:01:30.410 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:01:30.413 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:01:30.422 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:01 /home/ubuntu/cephtest 2024-03-11T22:01:30.424 INFO:teuthology.orchestra.run.smithi176.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:01 /home/ubuntu/cephtest 2024-03-11T22:01:30.441 INFO:teuthology.orchestra.run.smithi189.stdout: 265940 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:01 /home/ubuntu/cephtest 2024-03-11T22:01:30.442 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:01:30.453 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:01:30.548 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:01:30.575 DEBUG:teuthology.nuke:shortname: smithi151 2024-03-11T22:01:30.575 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:01:30.599 DEBUG:teuthology.nuke:shortname: smithi176 2024-03-11T22:01:30.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:01:30.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:40:27.191340', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2aPs54Jdwcrh3ud3AlUTKuZhEbLl+WksdmQQRZUmtOq5s+e63jHRhrCCoscBSwF3q3ymTyEK3SpVJDsJM86g0='} 2024-03-11T22:01:30.620 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:40:27.194908', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBmDhu7mQ5IWqHxzvPYXJmeiS61UydZAuhIrjYjsMNocCQG9SQu1V9ykcPbZ0mDV90xyGOpANXGLhq5d5D4d0RU='} 2024-03-11T22:01:30.624 DEBUG:teuthology.nuke:shortname: smithi189 2024-03-11T22:01:30.625 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:01:30.645 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591947', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:40:27.193184', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqMMaV3SbeaoIprq5WD33/YqRg8D24pJPwyQPRRX8Nv4a9GUzsBskHdgO7RvvSVoa8RX2UcW61aQTQSiAs3xK8='} 2024-03-11T22:01:30.653 INFO:teuthology.orchestra.console.smithi151:Power off 2024-03-11T22:01:30.654 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:01:30.678 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-03-11T22:01:30.678 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:30.697 INFO:teuthology.orchestra.console.smithi176:Power off 2024-03-11T22:01:30.697 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:01:30.706 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-11T22:01:30.719 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-03-11T22:01:30.719 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:30.735 INFO:teuthology.orchestra.console.smithi189:Power off 2024-03-11T22:01:30.735 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:01:30.744 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-11T22:01:30.759 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-03-11T22:01:30.759 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:30.781 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-11T22:01:34.706 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:34.746 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:34.782 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:34.815 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-11T22:01:34.855 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-11T22:01:34.893 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-11T22:01:38.818 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:38.856 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:38.895 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:38.929 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-11T22:01:38.967 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-11T22:01:39.003 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-03-11T22:01:39.104 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-03-11T22:01:42.930 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:42.968 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:01:43.040 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-03-11T22:01:43.078 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-03-11T22:01:43.141 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-03-11T22:01:43.179 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-03-11T22:01:43.283 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} duration: 704.940043926239 failure_reason: '"2024-03-11T21:57:11.739599+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:01:43.283 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:01:43.371 INFO:teuthology.run:FAIL