2024-03-11T21:04:41.729 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T21:04:41.729 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} --verbose --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800 -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/orig.config.yaml 2024-03-11T21:04:41.757 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:04:41.837 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800 branch: main description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591800' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLjmPxKzqo5hW0t7uLGuSiT1cdqGWUjCaULDOw7MY7BoFDoFDFVZ1wf1QRkCm96r/HlE2jCalpYPDGENTJsQ+Eo= smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFaTxXxg/pq5ZBgqILyMobvDvddS5eTBGliaLkKLYgwl9pfk0K0+xI7/SmR1UNrnwa74q/6Iu6Ec4qTuIDz5uHU= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7MdCpn/e6IhZZobHWhkTFb6n9x/GDhgF6p8CHFyvpp8Re9PqSahYU2win+rD6ewfuy1NPNsIxWeecTRxmuMi4= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T21:04:41.837 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-11T21:04:41.838 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-11T21:04:41.838 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T21:04:41.840 INFO:teuthology.task.internal:Checking packages... 2024-03-11T21:04:41.864 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-11T21:04:41.864 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T21:04:41.865 INFO:teuthology.packaging:ref: None 2024-03-11T21:04:41.865 INFO:teuthology.packaging:tag: None 2024-03-11T21:04:41.865 INFO:teuthology.packaging:branch: main 2024-03-11T21:04:41.865 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:04:41.865 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-11T21:04:42.117 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-11T21:04:42.118 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T21:04:42.125 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T21:04:42.125 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T21:04:42.131 INFO:teuthology.task.internal:Saving configuration 2024-03-11T21:04:42.147 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T21:04:42.153 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:04:42.179 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:35.959325', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLjmPxKzqo5hW0t7uLGuSiT1cdqGWUjCaULDOw7MY7BoFDoFDFVZ1wf1QRkCm96r/HlE2jCalpYPDGENTJsQ+Eo='} 2024-03-11T21:04:42.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:35.960748', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFaTxXxg/pq5ZBgqILyMobvDvddS5eTBGliaLkKLYgwl9pfk0K0+xI7/SmR1UNrnwa74q/6Iu6Ec4qTuIDz5uHU='} 2024-03-11T21:04:42.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:35.957824', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7MdCpn/e6IhZZobHWhkTFb6n9x/GDhgF6p8CHFyvpp8Re9PqSahYU2win+rD6ewfuy1NPNsIxWeecTRxmuMi4='} 2024-03-11T21:04:42.225 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T21:04:42.232 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T21:04:42.232 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T21:04:42.232 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-11T21:04:42.232 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T21:04:42.339 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcd06edaf70>, signals=[15]) 2024-03-11T21:04:42.339 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T21:04:42.382 INFO:teuthology.task.internal:Opening connections... 2024-03-11T21:04:42.383 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-03-11T21:04:42.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:04:42.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-03-11T21:04:42.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:04:42.604 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-11T21:04:42.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:04:42.691 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T21:04:42.698 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-03-11T21:04:42.716 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-03-11T21:04:42.716 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:04:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:04:42.775 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:04:42.775 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:04:42.775 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:04:42.775 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:04:42.775 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-03-11T21:04:42.800 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-03-11T21:04:42.817 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-03-11T21:04:42.817 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-03-11T21:04:42.873 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-03-11T21:04:42.873 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:04:42.874 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:04:42.875 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-03-11T21:04:42.916 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-11T21:04:42.934 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-11T21:04:42.935 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:04:42.989 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:04:42.990 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:04:42.990 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:04:42.990 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-11T21:04:43.016 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T21:04:43.026 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T21:04:43.031 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T21:04:43.032 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:04:43.034 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:04:43.037 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:04:43.054 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T21:04:43.060 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T21:04:43.060 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:04:43.092 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:04:43.094 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:04:43.120 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T21:04:43.202 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T21:04:43.216 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T21:04:43.216 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T21:04:43.217 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:04:43.217 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:04:43.218 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:04:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.237 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-11T21:04:43.237 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-03-11T21:04:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:04:43.239 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-11T21:04:43.293 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:04:43.293 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-361.el9.x86_64 2024-03-11T21:04:43.294 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-03-11T21:04:43.295 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:04:43.295 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-361.el9.x86_64 2024-03-11T21:04:43.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-03-11T21:04:43.297 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:04:43.297 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-361.el9.x86_64 2024-03-11T21:04:43.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-03-11T21:04:45.378 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T21:04:45.431 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:04:45.562 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T21:04:46.895 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-11T21:04:46.915 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-11T21:04:47.004 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-11T21:04:47.697 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-11T21:04:47.803 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-11T21:04:49.094 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-03-11T21:04:49.837 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-11T21:04:50.573 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-11T21:04:54.097 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-11T21:04:55.190 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-11T21:04:55.898 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 15 kB 00:00 2024-03-11T21:04:56.649 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 15 kB 00:00 2024-03-11T21:04:56.799 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-11T21:05:01.665 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-11T21:05:03.872 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:05:03.872 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:05:03.969 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-11T21:05:03.979 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-11T21:05:03.979 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-11T21:05:03.979 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-11T21:05:03.980 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-11T21:05:03.981 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-03-11T21:05:03.981 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:05:03.981 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 100 M 2024-03-11T21:05:03.981 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 123 M 2024-03-11T21:05:03.981 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-11T21:05:05.833 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.5 MB/s | 6.0 MB 00:01 2024-03-11T21:05:07.692 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 5.8 MB/s | 21 MB 00:03 2024-03-11T21:05:09.275 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 7.6 MB/s | 39 MB 00:05 2024-03-11T21:05:11.585 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:14 2024-03-11T21:05:16.799 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-11T21:05:17.660 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 2.8 MB/s | 33 MB 00:11 2024-03-11T21:05:17.660 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:05:17.660 INFO:teuthology.orchestra.run.smithi204.stdout:Total 7.3 MB/s | 100 MB 00:13 2024-03-11T21:05:18.546 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-11T21:05:18.891 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-11T21:05:18.891 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-11T21:05:19.173 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:05:19.173 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:05:19.284 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-11T21:05:19.289 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-11T21:05:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-03-11T21:05:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:05:19.291 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 100 M 2024-03-11T21:05:19.291 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 123 M 2024-03-11T21:05:19.291 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-11T21:05:19.725 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-03-11T21:05:20.244 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-11T21:05:20.244 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-11T21:05:20.959 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 27 MB/s | 33 MB 00:01 2024-03-11T21:05:21.242 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-03-11T21:05:23.224 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-11T21:05:23.509 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 9.4 MB/s | 39 MB 00:04 2024-03-11T21:05:23.509 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:05:23.509 INFO:teuthology.orchestra.run.smithi017.stdout:Total 24 MB/s | 100 MB 00:04 2024-03-11T21:05:23.574 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 578 kB/s | 19 MB 00:33 2024-03-11T21:05:23.900 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:05:24.026 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:05:24.434 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-11T21:05:24.839 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-11T21:05:24.840 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-11T21:05:26.285 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-11T21:05:26.285 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-11T21:05:26.786 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:05:26.964 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:05:28.758 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-11T21:05:29.160 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-11T21:05:29.907 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:05:30.026 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:05:30.117 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 15 kB 00:00 2024-03-11T21:05:31.735 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:05:32.149 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:05:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:05:33.153 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:05:33.393 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 21 MB 00:03 2024-03-11T21:05:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:05:38.258 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:05:38.382 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:05:38.468 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-11T21:05:40.758 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:05:40.759 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:05:40.844 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-11T21:05:40.848 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-11T21:05:40.848 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-03-11T21:05:40.849 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:05:40.850 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 100 M 2024-03-11T21:05:40.850 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 123 M 2024-03-11T21:05:40.850 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-11T21:05:43.513 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:05:51.310 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 594 kB/s | 6.0 MB 00:10 2024-03-11T21:06:06.195 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:06.196 INFO:teuthology.orchestra.run.smithi204.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:06:06.196 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:06:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:06.771 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:06.771 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:06:06.771 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:06:06.771 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:06:07.190 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:06:07.191 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-11T21:06:07.698 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:06:07.842 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:06:07.842 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:06:07.843 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:06:07.843 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:06:07.843 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:06:07.843 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T21:06:07.843 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:06:07.843 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:06:07.963 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:06:07.963 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:06:07.963 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-03-11T21:06:08.003 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 68.el9 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8264254 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:06:08.004 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:06:08.005 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:06:08.006 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:06:08.006 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:06:08.007 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:06:08.626 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-03-11T21:06:10.069 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:06:10.109 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-03-11T21:06:10.112 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-11T21:06:10.127 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.jkkES2bc0g 2024-03-11T21:06:10.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jkkES2bc0g 2024-03-11T21:06:10.195 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.jkkES2bc0g 2024-03-11T21:06:10.330 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.jkkES2bc0g is 6KB 2024-03-11T21:06:10.379 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.jkkES2bc0g 2024-03-11T21:06:10.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:06:10.459 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:06:10.459 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:06:10.459 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:06:10.459 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:06:10.460 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:06:10.613 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-03-11T21:06:13.125 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 664 kB/s | 21 MB 00:32 2024-03-11T21:06:14.140 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:14.140 INFO:teuthology.orchestra.run.smithi017.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:06:14.140 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:06:14.228 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:14.690 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:14.691 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:06:14.691 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:06:14.691 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:06:15.146 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:15.146 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:06:15.146 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-11T21:06:15.147 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:06:15.147 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:06:15.147 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:06:15.147 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:06:15.147 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:06:15.147 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-11T21:06:15.693 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:06:15.848 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:06:15.849 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:06:15.849 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:06:15.849 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:06:15.849 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:06:15.849 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T21:06:15.849 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:06:15.849 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:06:15.976 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:06:15.976 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:06:15.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 68.el9 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8264254 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:06:16.025 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:06:16.026 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:06:16.027 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:06:16.027 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:06:16.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:06:16.597 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-03-11T21:06:17.993 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:06:18.020 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-03-11T21:06:18.023 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-03-11T21:06:18.038 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.y21y4UCC7z 2024-03-11T21:06:18.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y21y4UCC7z 2024-03-11T21:06:18.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.y21y4UCC7z 2024-03-11T21:06:18.257 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.y21y4UCC7z is 6KB 2024-03-11T21:06:18.305 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.y21y4UCC7z 2024-03-11T21:06:18.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:06:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:06:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:06:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:06:18.390 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:06:18.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:06:18.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-03-11T21:06:34.656 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 750 kB/s | 39 MB 00:53 2024-03-11T21:06:39.172 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 715 kB/s | 33 MB 00:47 2024-03-11T21:06:39.172 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:06:39.172 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.7 MB/s | 100 MB 00:58 2024-03-11T21:06:40.091 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-11T21:06:40.527 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-11T21:06:40.528 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-11T21:06:40.616 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T21:06:40.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:06:40.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:06:41.930 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-11T21:06:41.930 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-11T21:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-11T21:06:45.454 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:06:45.614 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:06:48.521 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:06:48.544 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:06:48.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-11T21:06:48.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:06:48.698 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:06:53.823 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:06:53.937 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:06:58.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-11T21:06:59.090 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:06.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-03-11T21:07:07.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:07:07.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:07:11.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-11T21:07:15.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-11T21:07:15.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:07:18.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-03-11T21:07:23.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:07:23.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:07:29.625 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:29.626 INFO:teuthology.orchestra.run.smithi001.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:07:29.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:07:29.712 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:30.164 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:30.164 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:07:30.164 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:07:30.164 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:07:30.589 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:07:30.590 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:07:30.590 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-11T21:07:30.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-11T21:07:30.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:07:31.076 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:07:31.228 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:07:31.229 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:07:31.229 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:07:31.229 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:07:31.229 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:07:31.229 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T21:07:31.229 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:07:31.229 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:07:31.356 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:07:31.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:07:31.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-03-11T21:07:31.399 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:07:31.400 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:07:31.401 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:07:31.402 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:07:31.402 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:07:31.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:07:32.038 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-03-11T21:07:33.423 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:07:33.456 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-03-11T21:07:33.459 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-03-11T21:07:33.473 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.lL5hpbNaKR 2024-03-11T21:07:33.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lL5hpbNaKR 2024-03-11T21:07:33.546 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.lL5hpbNaKR 2024-03-11T21:07:33.689 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.lL5hpbNaKR is 6KB 2024-03-11T21:07:33.737 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.lL5hpbNaKR 2024-03-11T21:07:33.752 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:07:33.828 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:07:33.828 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:07:33.828 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:07:33.829 INFO:teuthology.orchestra.run.smithi001.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:07:33.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:07:33.981 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-03-11T21:07:41.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-11T21:07:56.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:07:56.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:07:57.089 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-11T21:07:57.508 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:07:57.508 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:07:57.509 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-11T21:07:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:07:57.523 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:07:57.523 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:07:57.523 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:07:58.524 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T21:07:58.524 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:07:58.599 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-03-11T21:07:58.639 DEBUG:teuthology.parallel:result is None 2024-03-11T21:08:02.750 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-03-11T21:08:03.131 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-11T21:08:03.131 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:08:03.131 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-03-11T21:08:03.145 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:08:03.146 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:08:03.146 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:08:03.146 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:08:03.985 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:08:03.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-11T21:08:03.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:08:04.147 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:08:04.147 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:08:04.240 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-03-11T21:08:04.277 DEBUG:teuthology.parallel:result is None 2024-03-11T21:08:22.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-03-11T21:08:31.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-11T21:08:31.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:08:34.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-03-11T21:08:46.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-11T21:08:46.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:18.327 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-11T21:09:18.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-11T21:09:18.715 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:09:18.715 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-03-11T21:09:18.730 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:09:18.730 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:09:18.731 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:09:18.731 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:09:19.732 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:09:19.732 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:09:19.800 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-03-11T21:09:19.836 DEBUG:teuthology.parallel:result is None 2024-03-11T21:09:19.837 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:09:19.844 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:09:19.844 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:09:19.847 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:09:19.850 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:09:19.869 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:09:19.940 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:09:19.948 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:09:19.949 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:09:19.951 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:09:19.953 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:09:19.983 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:09:19.989 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:09:19.989 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:09:20.020 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:09:20.023 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:09:20.051 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:20.059 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:20.065 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:20.073 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:20.073 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:20.087 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:20.089 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:09:20.096 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:09:20.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:09:20.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:09:20.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:09:20.148 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:09:20.157 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:09:20.158 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:09:20.180 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:09:20.187 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:09:20.203 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:20.279 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:20.328 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:20.413 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:20.457 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:09:20.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:09:20.527 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:20.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:20.625 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:20.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:20.754 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:09:20.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:09:20.824 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:20.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:20.933 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:20.977 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:21.021 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:09:21.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:09:21.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-03-11T21:09:21.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-03-11T21:09:21.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-11T21:09:21.137 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:09:21.149 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:09:21.199 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:09:21.654 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:09:21.662 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:09:21.662 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:09:21.671 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:09:21.680 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T21:09:21.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-03-11T21:09:21.740 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-03-11T21:09:21.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-03-11T21:09:21.790 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-03-11T21:09:21.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-03-11T21:09:21.863 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-03-11T21:09:21.864 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:09:21.865 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-03-11T21:09:21.901 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-03-11T21:09:21.902 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-03-11T21:09:21.925 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-03-11T21:09:21.926 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-03-11T21:09:21.951 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-03-11T21:09:21.951 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-03-11T21:09:21.952 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:09:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:21.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:09:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.010 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:09:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:22.038 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:09:22.038 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:09:22.070 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:09:22.102 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:09:22.145 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:09:22.155 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:09:22.175 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:09:22.176 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi017.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:13:40.233 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-11T21:13:40.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-11T21:13:40.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:13:40.325 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-03-11T21:13:40.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-03-11T21:13:40.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-11T21:13:40.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:13:40.489 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-03-11T21:13:40.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-11T21:13:40.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:13:40.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:13:40.655 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-11T21:13:40.745 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-11T21:13:40.746 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:13:40.756 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:13:40.756 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:13:40.756 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:13:40.759 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:13:40.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:13:40.761 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:13:40.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:13:40.790 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:13:40.792 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:13:40.807 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:13:40.810 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:13:40.818 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:13:40.835 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:13:40.837 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-03-11T21:13:40.840 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-03-11T21:13:40.851 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-03-11T21:13:40.854 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-03-11T21:13:40.865 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-03-11T21:13:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:13:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:13:40.879 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-03-11T21:13:40.884 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:13:40.888 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:13:40.896 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:13:40.912 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:13:40.936 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:13:40.937 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:13:40.938 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:13:40.939 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-11T21:13:40.939 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.939 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.939 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.939 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.940 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:13:40.940 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-03-11T21:13:40.940 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.940 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.940 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.940 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.963 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:13:40.967 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:13:40.967 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-11T21:13:40.967 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.967 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.967 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.968 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:13:40.968 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T21:13:40.977 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T21:13:40.977 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi001.front.sepia.ceph.com 2024-03-11T21:13:40.977 DEBUG:teuthology.orchestra.run.smithi001:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:13:40.978 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi017.front.sepia.ceph.com 2024-03-11T21:13:40.978 DEBUG:teuthology.orchestra.run.smithi017:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:13:40.978 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi204.front.sepia.ceph.com 2024-03-11T21:13:40.978 DEBUG:teuthology.orchestra.run.smithi204:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:13:41.568 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 09:12:53 PM UTC. 2024-03-11T21:13:41.576 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:50 ago on Mon 11 Mar 2024 09:12:51 PM UTC. 2024-03-11T21:13:41.594 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:53 ago on Mon 11 Mar 2024 09:12:48 PM UTC. 2024-03-11T21:13:41.720 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-11T21:13:41.727 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-03-11T21:13:41.728 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:13:41.739 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-11T21:13:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-03-11T21:13:41.739 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-11T21:13:41.739 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-03-11T21:13:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:13:41.739 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 3.1 M 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 9.8 M 2024-03-11T21:13:41.740 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-11T21:13:41.741 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-11T21:13:41.743 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.1 M 2024-03-11T21:13:41.743 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 9.8 M 2024-03-11T21:13:41.743 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-03-11T21:13:41.751 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-11T21:13:41.751 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-11T21:13:41.751 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-11T21:13:41.751 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-11T21:13:41.751 INFO:teuthology.orchestra.run.smithi204.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:13:41.752 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:13:41.752 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-11T21:13:41.752 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-11T21:13:41.752 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-11T21:13:41.752 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:13:41.757 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.1 M 2024-03-11T21:13:41.757 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 9.8 M 2024-03-11T21:13:41.757 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-11T21:13:42.294 INFO:teuthology.orchestra.run.smithi204.stdout:runc-1.1.12-2.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-03-11T21:13:42.294 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:13:42.295 INFO:teuthology.orchestra.run.smithi204.stdout:Total 5.8 MB/s | 3.1 MB 00:00 2024-03-11T21:13:42.330 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-11T21:13:42.336 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-11T21:13:42.336 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-11T21:13:42.415 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-11T21:13:42.415 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-11T21:13:42.634 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-11T21:13:42.739 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:42.774 INFO:teuthology.orchestra.run.smithi001.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.1 MB/s | 3.1 MB 00:00 2024-03-11T21:13:42.775 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:13:42.775 INFO:teuthology.orchestra.run.smithi001.stdout:Total 3.0 MB/s | 3.1 MB 00:01 2024-03-11T21:13:42.812 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-03-11T21:13:42.818 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-03-11T21:13:42.818 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-03-11T21:13:42.901 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-03-11T21:13:42.901 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-03-11T21:13:43.119 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-03-11T21:13:43.132 INFO:teuthology.orchestra.run.smithi017.stdout:runc-1.1.12-2.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-11T21:13:43.133 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:13:43.133 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.2 MB/s | 3.1 MB 00:01 2024-03-11T21:13:43.171 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-11T21:13:43.178 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-11T21:13:43.178 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-11T21:13:43.221 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:43.262 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-11T21:13:43.262 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-11T21:13:43.309 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-11T21:13:43.584 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:43.651 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:43.651 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:13:43.651 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-11T21:13:43.651 INFO:teuthology.orchestra.run.smithi204.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:13:43.651 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:13:43.651 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-11T21:13:43.789 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:43.883 DEBUG:teuthology.parallel:result is None 2024-03-11T21:13:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:13:44.092 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-03-11T21:13:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:13:44.092 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:13:44.092 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-03-11T21:13:44.131 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:44.325 DEBUG:teuthology.parallel:result is None 2024-03-11T21:13:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:13:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:13:44.418 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-11T21:13:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:13:44.418 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-11T21:13:44.418 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-11T21:13:44.646 DEBUG:teuthology.parallel:result is None 2024-03-11T21:13:44.647 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T21:13:44.658 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T21:13:44.658 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:13:44.658 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:13:44.674 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:13:44.674 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-03-11T21:13:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:13:44.729 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:44.729 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-11T21:13:44.730 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:44.730 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:44.730 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:13:39.210036316 +0000 2024-03-11T21:13:44.730 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:15.458298759 +0000 2024-03-11T21:13:44.730 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:15.458298759 +0000 2024-03-11T21:13:44.730 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:15.458298759 +0000 2024-03-11T21:13:44.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:13:44.796 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:13:44.796 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:13:44.796 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.00010361 s, 4.9 MB/s 2024-03-11T21:13:44.797 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:13:44.853 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-03-11T21:13:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:13:44.908 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:13:39.210036316 +0000 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:15.797295039 +0000 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:15.797295039 +0000 2024-03-11T21:13:44.909 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:15.797295039 +0000 2024-03-11T21:13:44.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:13:44.974 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:13:44.974 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:13:44.975 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.9955e-05 s, 5.7 MB/s 2024-03-11T21:13:44.976 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:13:45.031 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-03-11T21:13:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:13:45.085 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:45.085 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T21:13:45.086 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:45.086 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:45.086 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:13:39.210036316 +0000 2024-03-11T21:13:45.086 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:16.121291484 +0000 2024-03-11T21:13:45.086 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:16.121291484 +0000 2024-03-11T21:13:45.086 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:16.121291484 +0000 2024-03-11T21:13:45.086 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:13:45.150 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:13:45.150 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:13:45.150 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.3394e-05 s, 5.5 MB/s 2024-03-11T21:13:45.151 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:13:45.206 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-03-11T21:13:45.261 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:13:45.261 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:13:39.210036316 +0000 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:16.456287808 +0000 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:16.456287808 +0000 2024-03-11T21:13:45.262 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:16.456287808 +0000 2024-03-11T21:13:45.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:13:45.326 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:13:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:13:45.327 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.5103e-05 s, 5.4 MB/s 2024-03-11T21:13:45.328 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:13:45.383 DEBUG:teuthology.orchestra.run.smithi001:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:13:45.536 INFO:teuthology.orchestra.run.smithi001.stdout:loop 2024-03-11T21:13:45.537 INFO:tasks.nvme_loop:Connecting nvme_loop smithi001:/dev/vg_nvme/lv_1... 2024-03-11T21:13:45.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:13:45.574 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-11T21:13:45.612 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:13:45.648 INFO:tasks.nvme_loop:Connecting nvme_loop smithi001:/dev/vg_nvme/lv_2... 2024-03-11T21:13:45.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:13:45.728 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-11T21:13:45.767 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:13:45.797 INFO:tasks.nvme_loop:Connecting nvme_loop smithi001:/dev/vg_nvme/lv_3... 2024-03-11T21:13:45.797 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:13:45.876 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-11T21:13:45.914 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:13:45.943 INFO:tasks.nvme_loop:Connecting nvme_loop smithi001:/dev/vg_nvme/lv_4... 2024-03-11T21:13:45.944 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:13:46.024 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-03-11T21:13:46.063 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:13:46.092 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:13:46.092 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:13:46.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo nvme list 2024-03-11T21:13:46.213 INFO:teuthology.orchestra.run.smithi001.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:13:46.213 INFO:teuthology.orchestra.run.smithi001.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:13:46.213 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/nvme0n1 PHMB7513000D480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-11T21:13:46.213 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/nvme1n1 43d80df6bfb4a8ce5548 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:46.213 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/nvme2n1 dee3708d1c6d56c7b8c4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:46.213 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/nvme3n1 bb4c69966ceb20c7725a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:46.214 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/nvme4n1 ceb1424241b6263cc77a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:46.214 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:13:46.215 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:13:46.215 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/scratch_devs 2024-03-11T21:13:46.281 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:13:46.281 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:13:46.300 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:13:46.300 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:13:39.175255559 +0000 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:17.872490826 +0000 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:17.872490826 +0000 2024-03-11T21:13:46.358 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:17.872490826 +0000 2024-03-11T21:13:46.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:13:46.427 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:13:46.428 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:13:46.428 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000113091 s, 4.5 MB/s 2024-03-11T21:13:46.429 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:13:46.487 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:13:39.175255559 +0000 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:18.213487091 +0000 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:18.213487091 +0000 2024-03-11T21:13:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:18.213487091 +0000 2024-03-11T21:13:46.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:13:46.612 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:13:46.612 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:13:46.612 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000111691 s, 4.6 MB/s 2024-03-11T21:13:46.613 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:13:46.670 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-03-11T21:13:46.726 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:13:46.726 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:46.726 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T21:13:46.726 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:46.726 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:46.727 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:13:39.175255559 +0000 2024-03-11T21:13:46.727 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:18.560483291 +0000 2024-03-11T21:13:46.727 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:18.560483291 +0000 2024-03-11T21:13:46.727 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:18.560483291 +0000 2024-03-11T21:13:46.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:13:46.793 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:13:46.793 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:13:46.793 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000101064 s, 5.1 MB/s 2024-03-11T21:13:46.794 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:13:46.850 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-03-11T21:13:46.905 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:13:46.905 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:46.905 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T21:13:46.905 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:46.906 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:46.906 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:13:39.175255559 +0000 2024-03-11T21:13:46.906 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:18.884479742 +0000 2024-03-11T21:13:46.906 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:18.884479742 +0000 2024-03-11T21:13:46.906 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:18.884479742 +0000 2024-03-11T21:13:46.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:13:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:13:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:13:46.971 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000100064 s, 5.1 MB/s 2024-03-11T21:13:46.972 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:13:47.027 DEBUG:teuthology.orchestra.run.smithi017:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:13:47.169 INFO:teuthology.orchestra.run.smithi017.stdout:loop 2024-03-11T21:13:47.170 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_1... 2024-03-11T21:13:47.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:13:47.207 INFO:teuthology.orchestra.run.smithi017.stdout:1 2024-03-11T21:13:47.244 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:13:47.273 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_2... 2024-03-11T21:13:47.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:13:47.352 INFO:teuthology.orchestra.run.smithi017.stdout:1 2024-03-11T21:13:47.395 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:13:47.424 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_3... 2024-03-11T21:13:47.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:13:47.500 INFO:teuthology.orchestra.run.smithi017.stdout:1 2024-03-11T21:13:47.538 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:13:47.567 INFO:tasks.nvme_loop:Connecting nvme_loop smithi017:/dev/vg_nvme/lv_4... 2024-03-11T21:13:47.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:13:47.646 INFO:teuthology.orchestra.run.smithi017.stdout:1 2024-03-11T21:13:47.684 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:13:47.714 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:13:47.714 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:13:47.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo nvme list 2024-03-11T21:13:47.836 INFO:teuthology.orchestra.run.smithi017.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:13:47.836 INFO:teuthology.orchestra.run.smithi017.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:13:47.836 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme0n1 PHMB7473003Q480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-11T21:13:47.836 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme1n1 1954de3b0db8e35e0ad7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:47.836 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme2n1 11258fd07a1ec08c7132 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:47.837 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme3n1 c93fa56772d41125d7dc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:47.837 INFO:teuthology.orchestra.run.smithi017.stdout:/dev/nvme4n1 23ea537f8325c8a8025b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:47.837 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:13:47.838 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:13:47.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/scratch_devs 2024-03-11T21:13:47.905 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:13:47.905 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:13:47.924 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:13:47.925 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-11T21:13:47.982 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:13:47.982 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:13:39.200421613 +0000 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:13.225824010 +0000 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:13.225824010 +0000 2024-03-11T21:13:47.983 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:13.225824010 +0000 2024-03-11T21:13:47.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:13:48.053 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:13:48.053 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:13:48.053 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00027626 s, 1.9 MB/s 2024-03-11T21:13:48.054 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:13:48.111 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-11T21:13:48.168 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:13:48.168 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:48.168 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T21:13:48.169 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:48.169 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:48.169 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:13:39.200421613 +0000 2024-03-11T21:13:48.169 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:13.576818599 +0000 2024-03-11T21:13:48.169 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:13.576818599 +0000 2024-03-11T21:13:48.169 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:13.576818599 +0000 2024-03-11T21:13:48.169 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:13:48.236 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:13:48.236 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:13:48.237 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000167614 s, 3.1 MB/s 2024-03-11T21:13:48.238 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:13:48.294 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-11T21:13:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:13:48.350 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:13:39.200421613 +0000 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:13.921813279 +0000 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:13.921813279 +0000 2024-03-11T21:13:48.351 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:13.921813279 +0000 2024-03-11T21:13:48.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:13:48.417 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:13:48.417 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:13:48.417 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000234559 s, 2.2 MB/s 2024-03-11T21:13:48.419 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:13:48.476 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:13:39.200421613 +0000 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:14.264807991 +0000 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:14.264807991 +0000 2024-03-11T21:13:48.531 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:14.264807991 +0000 2024-03-11T21:13:48.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:13:48.597 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:13:48.597 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:13:48.597 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000207444 s, 2.5 MB/s 2024-03-11T21:13:48.598 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:13:48.653 DEBUG:teuthology.orchestra.run.smithi204:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:13:48.821 INFO:teuthology.orchestra.run.smithi204.stdout:loop 2024-03-11T21:13:48.822 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_1... 2024-03-11T21:13:48.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:13:48.860 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-11T21:13:48.898 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:13:48.927 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_2... 2024-03-11T21:13:48.927 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:13:49.007 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-11T21:13:49.046 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:13:49.082 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_3... 2024-03-11T21:13:49.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:13:49.161 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-11T21:13:49.199 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:13:49.229 INFO:tasks.nvme_loop:Connecting nvme_loop smithi204:/dev/vg_nvme/lv_4... 2024-03-11T21:13:49.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:13:49.309 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-11T21:13:49.349 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:13:49.378 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:13:49.378 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:13:49.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme list 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme0n1 CVFT623300GD400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme1n1 ffc4e8a5307b0245fa3d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme2n1 f25d6f4942282c96e820 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme3n1 e22d56a2119ffbf430e3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:49.500 INFO:teuthology.orchestra.run.smithi204.stdout:/dev/nvme4n1 1496ef3e468e76286588 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:13:49.501 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:13:49.501 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:13:49.501 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/scratch_devs 2024-03-11T21:13:49.568 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T21:13:49.655 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-11T21:13:49.655 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:13:49.656 INFO:tasks.cephadm:Cluster fsid is 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:13:49.656 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T21:13:49.656 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.1', 'mon.b': '172.21.15.17', 'mon.c': '172.21.15.204'} 2024-03-11T21:13:49.656 INFO:tasks.cephadm:First mon is mon.a on smithi001 2024-03-11T21:13:49.656 INFO:tasks.cephadm:First mgr is y 2024-03-11T21:13:49.656 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T21:13:49.656 DEBUG:teuthology.orchestra.run.smithi001:> sudo hostname $(hostname -s) 2024-03-11T21:13:49.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo hostname $(hostname -s) 2024-03-11T21:13:49.722 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-03-11T21:13:49.750 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T21:13:49.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:13:50.052 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-11T21:13:50.190 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:13:50.192 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm 2024-03-11T21:13:50.192 DEBUG:teuthology.orchestra.run.smithi001:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:50.620 INFO:teuthology.orchestra.run.smithi001.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 21:13 /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:50.620 DEBUG:teuthology.orchestra.run.smithi017:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:51.418 INFO:teuthology.orchestra.run.smithi017.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 21:13 /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:51.418 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:51.939 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 21:13 /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:51.939 DEBUG:teuthology.orchestra.run.smithi001:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:51.957 DEBUG:teuthology.orchestra.run.smithi017:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:51.976 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:13:52.010 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-11T21:13:52.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:13:52.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:13:52.020 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:13:52.810 INFO:teuthology.orchestra.run.smithi001.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:13:52.859 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:13:52.939 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:14:53.485 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-03-11T21:14:53.486 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:14:53.486 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:14:53.486 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-03-11T21:14:53.486 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:14:53.486 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-03-11T21:14:53.486 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-03-11T21:14:54.095 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-11T21:14:54.096 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:14:54.096 INFO:teuthology.orchestra.run.smithi017.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:14:54.096 INFO:teuthology.orchestra.run.smithi017.stdout: "repo_digests": [ 2024-03-11T21:14:54.096 INFO:teuthology.orchestra.run.smithi017.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:14:54.096 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-03-11T21:14:54.096 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout: "repo_digests": [ 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-03-11T21:14:54.875 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-03-11T21:14:54.899 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph 2024-03-11T21:14:54.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2024-03-11T21:14:54.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-03-11T21:14:55.017 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /etc/ceph 2024-03-11T21:14:55.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /etc/ceph 2024-03-11T21:14:55.073 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-03-11T21:14:55.105 INFO:tasks.cephadm:Writing seed config... 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T21:14:55.107 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T21:14:55.108 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T21:14:55.108 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:14:55.108 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T21:14:55.122 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 415704e2-dfec-11ee-95c8-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T21:14:55.123 DEBUG:teuthology.orchestra.run.smithi001:mon.a> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T21:14:55.165 DEBUG:teuthology.orchestra.run.smithi001:mgr.y> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T21:14:55.207 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T21:14:55.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.1 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:14:55.402 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:14:55.402 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '415704e2-dfec-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.1', '--skip-admin-label'] 2024-03-11T21:14:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:14:55.428 INFO:teuthology.orchestra.run.smithi001.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T21:14:55.428 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying podman|docker is present... 2024-03-11T21:14:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:14:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying lvm2 is present... 2024-03-11T21:14:55.451 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying time synchronization is in place... 2024-03-11T21:14:55.458 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:14:55.458 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:14:55.463 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:14:55.463 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-03-11T21:14:55.469 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-03-11T21:14:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-03-11T21:14:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:14:55.475 INFO:teuthology.orchestra.run.smithi001.stdout:Repeating the final host check... 2024-03-11T21:14:55.498 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:14:55.498 INFO:teuthology.orchestra.run.smithi001.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T21:14:55.499 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl is present 2024-03-11T21:14:55.499 INFO:teuthology.orchestra.run.smithi001.stdout:lvcreate is present 2024-03-11T21:14:55.505 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:14:55.505 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:14:55.510 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:14:55.510 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-03-11T21:14:55.516 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-03-11T21:14:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-03-11T21:14:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:14:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:Host looks OK 2024-03-11T21:14:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:Cluster fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:14:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:Acquiring lock 139947169183104 on /run/cephadm/415704e2-dfec-11ee-95c8-87774f69a715.lock 2024-03-11T21:14:55.523 INFO:teuthology.orchestra.run.smithi001.stdout:Lock 139947169183104 acquired on /run/cephadm/415704e2-dfec-11ee-95c8-87774f69a715.lock 2024-03-11T21:14:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 3300 ... 2024-03-11T21:14:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 6789 ... 2024-03-11T21:14:55.524 INFO:teuthology.orchestra.run.smithi001.stdout:Base mon IP(s) is [172.21.15.1:3300, 172.21.15.1:6789], mon addrv is [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-03-11T21:14:55.526 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.1 metric 100 2024-03-11T21:14:55.526 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.1 metric 100 2024-03-11T21:14:55.529 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T21:14:55.529 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T21:14:55.529 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T21:14:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T21:14:55.532 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T21:14:55.532 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:14:55.532 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T21:14:55.532 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-03-11T21:14:55.532 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:14:55.533 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-03-11T21:14:55.533 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-03-11T21:14:55.533 INFO:teuthology.orchestra.run.smithi001.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T21:14:55.533 INFO:teuthology.orchestra.run.smithi001.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T21:14:55.533 INFO:teuthology.orchestra.run.smithi001.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T21:14:55.995 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T21:14:57.091 INFO:teuthology.orchestra.run.smithi001.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:14:57.091 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:14:57.091 INFO:teuthology.orchestra.run.smithi001.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T21:14:58.080 INFO:teuthology.orchestra.run.smithi001.stdout:stat: stdout 167 167 2024-03-11T21:14:58.080 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial keys... 2024-03-11T21:14:59.185 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQBSdO9lMtUbGxAAx6/eKtR6ELqtN80dqLuHhw== 2024-03-11T21:15:00.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQBTdO9lyeH/IRAApSuYaUDy/6y9HTDlzTns+Q== 2024-03-11T21:15:01.037 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQBUdO9lbfSwIBAAdem0hdhRXfxz6YFroKkLjA== 2024-03-11T21:15:01.037 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial monmap... 2024-03-11T21:15:02.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:15:02.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T21:15:02.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:02.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:15:02.118 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool for a [v2:172.21.15.1:3300,v1:172.21.15.1:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:15:02.119 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-03-11T21:15:02.119 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: set fsid to 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:02.119 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:15:02.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:15:02.119 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mon... 2024-03-11T21:15:03.173 INFO:teuthology.orchestra.run.smithi001.stdout:create mon.a on 2024-03-11T21:15:03.527 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T21:15:03.695 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-415704e2-dfec-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-415704e2-dfec-11ee-95c8-87774f69a715.target. 2024-03-11T21:15:03.695 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-415704e2-dfec-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-415704e2-dfec-11ee-95c8-87774f69a715.target. 2024-03-11T21:15:03.905 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:15:03.906 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T21:15:04.070 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-415704e2-dfec-11ee-95c8-87774f69a715.target.wants/ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-415704e2-dfec-11ee-95c8-87774f69a715@.service. 2024-03-11T21:15:04.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:04 smithi001 systemd[1]: Started Ceph mon.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:15:04.573 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:15:04.573 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-03-11T21:15:04.580 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:15:04.580 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-03-11T21:15:04.581 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-03-11T21:15:04.581 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:15:04.581 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon to start... 2024-03-11T21:15:04.581 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon... 2024-03-11T21:15:04.825 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:04 smithi001 ceph-mon[25283]: mkfs 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:04.825 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:04 smithi001 ceph-mon[25283]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:15:05.657 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:05 smithi001 ceph-mon[25283]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:15:05.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:05 smithi001 ceph-mon[25283]: monmap e1: 1 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:05.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:05 smithi001 ceph-mon[25283]: fsmap 2024-03-11T21:15:05.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:05 smithi001 ceph-mon[25283]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:15:05.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:05 smithi001 ceph-mon[25283]: mgrmap e1: no daemons active 2024-03-11T21:15:05.658 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:05 smithi001 ceph-mon[25283]: from='client.? 172.21.15.1:0/1469754911' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout id: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout services: 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558293s) 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T21:15:05.660 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout data: 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:mon is available 2024-03-11T21:15:05.661 INFO:teuthology.orchestra.run.smithi001.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T21:15:06.796 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:06 smithi001 ceph-mon[25283]: from='client.? 172.21.15.1:0/722654467' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:15:06.796 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:06 smithi001 ceph-mon[25283]: from='client.? 172.21.15.1:0/722654467' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:15:06.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:06.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:15:06.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:15:06.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:15:06.801 INFO:teuthology.orchestra.run.smithi001.stdout:Generating new minimal ceph.conf... 2024-03-11T21:15:07.842 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:07 smithi001 ceph-mon[25283]: from='client.? 172.21.15.1:0/1088001696' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:07.845 INFO:teuthology.orchestra.run.smithi001.stdout:Restarting the monitor... 2024-03-11T21:15:08.116 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:07 smithi001 systemd[1]: Stopping Ceph mon.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:15:08.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:08 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a[25260]: 2024-03-11T21:15:08.129+0000 7f2a2286b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:15:08.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:08 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a[25260]: 2024-03-11T21:15:08.129+0000 7f2a2286b700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T21:15:08.646 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:08 smithi001 podman[25617]: 2024-03-11 21:15:08.395324925 +0000 UTC m=+0.373925464 container died 2c3aad81472bf2c216cce3a3b6dc1d2358633a06da110afab02a9bd99a2f054c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:15:08.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:08 smithi001 podman[25617]: 2024-03-11 21:15:08.759524243 +0000 UTC m=+0.738124832 container cleanup 2c3aad81472bf2c216cce3a3b6dc1d2358633a06da110afab02a9bd99a2f054c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-11T21:15:08.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:08 smithi001 bash[25617]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:15:09.376 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:09 smithi001 podman[25635]: 2024-03-11 21:15:09.018652348 +0000 UTC m=+0.622090004 container remove 2c3aad81472bf2c216cce3a3b6dc1d2358633a06da110afab02a9bd99a2f054c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T21:15:09.376 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:09 smithi001 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T21:15:09.377 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:09 smithi001 systemd[1]: Stopped Ceph mon.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:15:09.377 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:09 smithi001 systemd[1]: Starting Ceph mon.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:15:09.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:09 smithi001 podman[25736]: 2024-03-11 21:15:09.29404078 +0000 UTC m=+0.017896513 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:15:10.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:10 smithi001 podman[25736]: 2024-03-11 21:15:10.529182819 +0000 UTC m=+1.253038597 container create 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:15:12.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 podman[25736]: 2024-03-11 21:15:12.299944766 +0000 UTC m=+3.023800499 container init 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T21:15:12.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 podman[25736]: 2024-03-11 21:15:12.306745062 +0000 UTC m=+3.030600804 container start 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:15:12.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:15:12.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T21:15:12.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: pidfile_write: ignore empty --pid-file 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: load: jerasure load: lrc 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Git sha 0 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: DB SUMMARY 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: DB Session ID: RN85CKCYY1DD620Q3AUR 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84593 ; 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:15:12.709 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.env: 0x55ce38d29b20 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.info_log: 0x55ce39cbf8e0 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.statistics: (nil) 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.use_fsync: 0 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:15:12.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.db_log_dir: 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.wal_dir: 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:15:12.711 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.write_buffer_manager: 0x55ce39c93400 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:15:12.712 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.unordered_write: 0 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.row_cache: None 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.wal_filter: None 2024-03-11T21:15:12.713 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:15:12.714 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.wal_compression: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:15:12.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:15:12.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_open_files: -1 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Compression algorithms supported: 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kZSTD supported: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:15:12.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.merge_operator: 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_filter: None 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ce39cbf9e0) 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:15:12.718 INFO:journalctl@ceph.mon.a.smithi001.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_type: 0 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: data_block_index_type: 0 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_shortening: 1 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: checksum: 4 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: no_block_cache: 0 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache: 0x55ce39cc5090 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_options: 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: capacity : 536870912 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: num_shard_bits : 4 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: strict_capacity_limit : 0 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_compressed: (nil) 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: persistent_cache: (nil) 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_size: 4096 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_size_deviation: 10 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_restart_interval: 16 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_block_restart_interval: 1 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: metadata_block_size: 4096 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: partition_filters: 0 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: use_delta_encoding: 1 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: filter_policy: bloomfilter 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: whole_key_filtering: 1 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: verify_compression: 0 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: format_version: 5 2024-03-11T21:15:12.720 INFO:journalctl@ceph.mon.a.smithi001.stdout: enable_index_compression: 1 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_align: 0 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout: max_auto_readahead_size: 262144 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout: prepopulate_block_cache: 0 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression: NoCompression 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.num_levels: 7 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:15:12.721 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:15:12.722 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:15:12.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:15:12.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:15:12.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.ttl: 2592000 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0ab2e4fc-3c48-46c3-bab1-70abdfae8e03 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191712338668, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191712339608, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79405, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73714, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710191712, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0ab2e4fc-3c48-46c3-bab1-70abdfae8e03", "db_session_id": "RN85CKCYY1DD620Q3AUR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:15:12.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191712339676, "job": 1, "event": "recovery_finished"} 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ce39db8000 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: DB pointer 0x55ce39da2000 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** DB Stats ** 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:15:12.728 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** Compaction Stats [default] ** 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: L0 2/0 81.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: Sum 2/0 81.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** Compaction Stats [default] ** 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:15:12.729 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative compaction: 0.00 GB write, 24.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval compaction: 0.00 GB write, 24.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Block cache BinnedLRUCache@0x55ce39cc5090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:15:12.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: starting mon.a rank 0 at public addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] at bind addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???) e1 preinit fsid 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).mds e1 new map 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).mds e1 print_map 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout: e1 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout: legacy client fscid: -1 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout: No filesystems configured 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T21:15:12.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:15:12.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: monmap e1: 1 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:15:12.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: fsmap 2024-03-11T21:15:12.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:15:12.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:12 smithi001 ceph-mon[25774]: mgrmap e1: no daemons active 2024-03-11T21:15:13.193 INFO:teuthology.orchestra.run.smithi001.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T21:15:13.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:13 smithi001 bash[25736]: 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f 2024-03-11T21:15:13.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:13 smithi001 systemd[1]: Started Ceph mon.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:15:17.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:17 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1129450555' entity='client.admin' 2024-03-11T21:15:18.326 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T21:15:18.346 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:15:18.347 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mgr... 2024-03-11T21:15:18.347 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T21:15:18.348 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T21:15:18.602 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:15:18.603 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T21:15:18.746 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-415704e2-dfec-11ee-95c8-87774f69a715.target.wants/ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-415704e2-dfec-11ee-95c8-87774f69a715@.service. 2024-03-11T21:15:19.024 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 podman[26020]: 2024-03-11 21:15:19.038440588 +0000 UTC m=+0.159781010 container create 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T21:15:19.317 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 podman[26020]: 2024-03-11 21:15:19.26563886 +0000 UTC m=+0.386979243 container init 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-11T21:15:19.318 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 podman[26020]: 2024-03-11 21:15:19.271317829 +0000 UTC m=+0.392658213 container start 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T21:15:19.333 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:15:19.334 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-03-11T21:15:19.342 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:15:19.342 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-03-11T21:15:19.342 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-03-11T21:15:19.342 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:15:19.352 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:15:19.352 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-03-11T21:15:19.360 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:15:19.360 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-03-11T21:15:19.360 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-03-11T21:15:19.360 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T21:15:19.360 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr to start... 2024-03-11T21:15:19.360 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr... 2024-03-11T21:15:19.584 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 bash[26020]: 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 2024-03-11T21:15:19.585 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 systemd[1]: Started Ceph mgr.y for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:15:19.846 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:19.842+0000 7f42d9a47200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:15:20.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:19 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2652515335' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:20.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:19 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:19.938+0000 7f42d9a47200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "415704e2-dfec-11ee-95c8-87774f69a715", 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:15:20.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:15:20.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:15:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:15:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:15:20.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:15:20.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:15:20.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:15:20.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:15:20.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:15:20.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:15:04.575645+0000", 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:20.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:20.545 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:15:20.545 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:15:20.545 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (1/15)... 2024-03-11T21:15:20.829 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:20 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:20.607+0000 7f42d9a47200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:15:20.830 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:20 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:20.701+0000 7f42d9a47200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:15:21.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:20 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:20.843+0000 7f42d9a47200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:15:21.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:20 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:20.939+0000 7f42d9a47200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:15:21.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:21 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:21.023+0000 7f42d9a47200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:15:21.571 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:21 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:21.289+0000 7f42d9a47200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:15:21.571 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:21 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:21.400+0000 7f42d9a47200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:15:21.572 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:21 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:21.488+0000 7f42d9a47200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:15:21.572 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:21 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:21.584+0000 7f42d9a47200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:15:21.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:21 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:21.768+0000 7f42d9a47200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:15:22.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:22 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:22.760+0000 7f42d9a47200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:15:22.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:22 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:22.896+0000 7f42d9a47200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:15:23.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:23 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1478606993' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:23.317 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:23 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:23.239+0000 7f42d9a47200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:15:23.624 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:23 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:23.330+0000 7f42d9a47200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:15:23.624 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:23 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:23.529+0000 7f42d9a47200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:15:23.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:23 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:23.716+0000 7f42d9a47200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:15:24.252 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:24 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:24.003+0000 7f42d9a47200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:15:24.252 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:24 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:24.265+0000 7f42d9a47200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:15:24.612 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:24 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:24.506+0000 7f42d9a47200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "415704e2-dfec-11ee-95c8-87774f69a715", 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:24.618 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:15:24.619 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:15:24.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:15:24.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:15:24.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:15:24.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:15:24.623 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:24.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:15:04.575645+0000", 2024-03-11T21:15:24.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:24.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:24.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:15:24.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:15:24.625 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (2/15)... 2024-03-11T21:15:25.668 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:25 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:25.334+0000 7f42d9a47200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:15:25.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:25 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:25.925+0000 7f42d9a47200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:15:26.298 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:26.025+0000 7f42d9a47200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:15:26.298 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:26.247+0000 7f42d9a47200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:15:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: Activating manager daemon y 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: mgrmap e2: y(active, starting, since 0.00396187s) 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: Manager daemon y is now available 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' 2024-03-11T21:15:26.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:26 smithi001 ceph-mon[25774]: from='mgr.14100 172.21.15.1:0/1166577351' entity='mgr.y' 2024-03-11T21:15:27.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:27 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/3336977920' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:27.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:27 smithi001 ceph-mon[25774]: mgrmap e3: y(active, since 1.00556s) 2024-03-11T21:15:27.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:27.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "415704e2-dfec-11ee-95c8-87774f69a715", 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:15:27.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:15:27.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:15:27.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:15:27.754 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:15:27.755 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:15:04.575645+0000", 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:15:27.756 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (3/15)... 2024-03-11T21:15:29.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:29 smithi001 ceph-mon[25774]: mgrmap e4: y(active, since 2s) 2024-03-11T21:15:30.873 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:30 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/3689601838' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "415704e2-dfec-11ee-95c8-87774f69a715", 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:15:31.139 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:15:31.140 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:15:31.141 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:15:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:15:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:15:31.144 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:15:31.145 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:15:04.575645+0000", 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:15:31.146 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:15:31.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:15:31.147 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:15:31.147 INFO:teuthology.orchestra.run.smithi001.stdout:mgr is available 2024-03-11T21:15:32.198 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:31 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1078950013' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:15:32.198 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:31 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1078950013' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:15:32.535 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:15:32.536 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:15:32.537 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:15:32.537 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling cephadm module... 2024-03-11T21:15:33.637 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:33 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2169664264' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T21:15:33.637 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:33 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ignoring --setuser ceph since I am not root 2024-03-11T21:15:33.637 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:33 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ignoring --setgroup ceph since I am not root 2024-03-11T21:15:34.167 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:33 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:33.925+0000 7f88d0314200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:15:34.167 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:34 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:34.024+0000 7f88d0314200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:15:34.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:34 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2169664264' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T21:15:34.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:34 smithi001 ceph-mon[25774]: mgrmap e5: y(active, since 7s) 2024-03-11T21:15:34.923 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:34 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:34.697+0000 7f88d0314200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:15:34.924 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:34 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:34.793+0000 7f88d0314200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:15:34.924 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:34 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:34.937+0000 7f88d0314200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:15:35.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.034+0000 7f88d0314200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:15:35.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.119+0000 7f88d0314200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:15:35.239 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:15:35.239 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T21:15:35.239 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:15:35.239 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:15:35.239 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:15:35.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:15:35.240 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-03-11T21:15:35.240 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 5... 2024-03-11T21:15:35.509 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2931334291' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:15:35.509 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.396+0000 7f88d0314200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:15:35.509 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.522+0000 7f88d0314200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:15:35.875 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.610+0000 7f88d0314200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:15:35.876 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.709+0000 7f88d0314200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:15:35.876 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:35 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:35.888+0000 7f88d0314200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:15:37.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:36 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:36.874+0000 7f88d0314200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:15:37.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:37 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:37.008+0000 7f88d0314200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:15:37.623 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:37 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:37.344+0000 7f88d0314200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:15:37.623 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:37 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:37.437+0000 7f88d0314200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:15:37.623 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:37 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:37.636+0000 7f88d0314200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:15:37.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:37 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:37.825+0000 7f88d0314200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:15:38.362 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:38 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:38.114+0000 7f88d0314200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:15:38.707 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:38 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:38.375+0000 7f88d0314200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:15:38.707 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:38 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:38.616+0000 7f88d0314200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:15:39.707 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:39 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:39.437+0000 7f88d0314200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:15:40.325 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:40.019+0000 7f88d0314200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:15:40.325 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:40.119+0000 7f88d0314200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:15:40.325 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:15:40.338+0000 7f88d0314200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:15:40.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: Active manager daemon y restarted 2024-03-11T21:15:40.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: Activating manager daemon y 2024-03-11T21:15:40.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T21:15:40.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: mgrmap e6: y(active, starting, since 0.0031668s) 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: Manager daemon y is now available 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:40.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:40 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: Found migration_current of "None". Setting to last migration. 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:41.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:41 smithi001 ceph-mon[25774]: mgrmap e7: y(active, since 1.00466s) 2024-03-11T21:15:41.919 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:15:41.919 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T21:15:41.919 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:15:41.919 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:15:41.920 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 5 is available 2024-03-11T21:15:41.920 INFO:teuthology.orchestra.run.smithi001.stdout:Setting orchestrator backend to cephadm... 2024-03-11T21:15:42.653 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:42 smithi001 ceph-mon[25774]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:15:42.653 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:42 smithi001 ceph-mon[25774]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:15:43.500 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:43 smithi001 ceph-mon[25774]: mgrmap e8: y(active, since 2s) 2024-03-11T21:15:43.500 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:43 smithi001 ceph-mon[25774]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:43.500 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:43 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:43.500 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:43 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:43.501 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:43 smithi001 ceph-mon[25774]: [11/Mar/2024:21:15:43] ENGINE Bus STARTING 2024-03-11T21:15:43.501 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:43 smithi001 ceph-mon[25774]: [11/Mar/2024:21:15:43] ENGINE Serving on http://172.21.15.1:8765 2024-03-11T21:15:44.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:44 smithi001 ceph-mon[25774]: [11/Mar/2024:21:15:43] ENGINE Serving on https://172.21.15.1:7150 2024-03-11T21:15:44.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:44 smithi001 ceph-mon[25774]: [11/Mar/2024:21:15:43] ENGINE Bus STARTED 2024-03-11T21:15:44.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:44 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:44.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:44 smithi001 ceph-mon[25774]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T21:15:44.463 INFO:teuthology.orchestra.run.smithi001.stdout:Generating ssh key... 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: Generating public/private rsa key pair. 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: Your identification has been saved in /tmp/tmp_6hof8uc/key. 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: Your public key has been saved in /tmp/tmp_6hof8uc/key.pub. 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: The key fingerprint is: 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: SHA256:os/V28Syc6/PelIhxWUMW9i/jdhxCVOjE5zzGLV+Ti4 ceph-415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: The key's randomart image is: 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: +---[RSA 3072]----+ 2024-03-11T21:15:45.609 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | .o=X+| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | B*+=| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | +O.o| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | ..=oo| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | . S + *=| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | . . . .. +=o| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | . . o o.E o| 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | o . .*o... | 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: | o oo+B+ | 2024-03-11T21:15:45.610 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:15:45 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: +----[SHA256]-----+ 2024-03-11T21:15:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:46 smithi001 ceph-mon[25774]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:46 smithi001 ceph-mon[25774]: Generating ssh key... 2024-03-11T21:15:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:46 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:46 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:47.075 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:15:47.075 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:15:47.076 INFO:teuthology.orchestra.run.smithi001.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T21:15:47.076 INFO:teuthology.orchestra.run.smithi001.stdout:Adding host smithi001... 2024-03-11T21:15:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:47 smithi001 ceph-mon[25774]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:48.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:48 smithi001 ceph-mon[25774]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:49.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:49 smithi001 ceph-mon[25774]: Deploying cephadm binary to smithi001 2024-03-11T21:15:52.732 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Added host 'smithi001' with addr '172.21.15.1' 2024-03-11T21:15:52.732 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying unmanaged mon service... 2024-03-11T21:15:53.442 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:53 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:53.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:53 smithi001 ceph-mon[25774]: Added host smithi001 2024-03-11T21:15:53.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:53 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:54.007 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T21:15:54.007 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying unmanaged mgr service... 2024-03-11T21:15:54.625 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:54 smithi001 ceph-mon[25774]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:54.625 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:54 smithi001 ceph-mon[25774]: Saving service mon spec with placement count:5 2024-03-11T21:15:54.625 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:54 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:56.183 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:55 smithi001 ceph-mon[25774]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:15:56.183 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:55 smithi001 ceph-mon[25774]: Saving service mgr spec with placement count:2 2024-03-11T21:15:56.183 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:55 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:56.185 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T21:15:57.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:56 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:57.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:56 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:15:57.976 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:57 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/4294142662' entity='client.admin' 2024-03-11T21:15:58.959 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling the dashboard module... 2024-03-11T21:15:59.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:59 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/3190029615' entity='client.admin' 2024-03-11T21:15:59.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:15:59 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:16:00.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:00 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/16648902' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T21:16:00.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:00 smithi001 ceph-mon[25774]: from='mgr.14120 172.21.15.1:0/3570518899' entity='mgr.y' 2024-03-11T21:16:00.708 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:00 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ignoring --setuser ceph since I am not root 2024-03-11T21:16:00.708 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:00 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ignoring --setgroup ceph since I am not root 2024-03-11T21:16:00.984 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:00 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:00.902+0000 7fad06aea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:16:00.984 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:00 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:00.997+0000 7fad06aea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:16:01.556 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:01 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/16648902' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T21:16:01.556 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:01 smithi001 ceph-mon[25774]: mgrmap e9: y(active, since 19s) 2024-03-11T21:16:01.898 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:01 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:01.664+0000 7fad06aea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:16:01.898 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:01 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:01.763+0000 7fad06aea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:16:01.898 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:01 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:01.911+0000 7fad06aea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:16:02.158 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.010+0000 7fad06aea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:16:02.158 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.097+0000 7fad06aea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:16:02.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1074768736' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:16:02.457 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.375+0000 7fad06aea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:16:02.839 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.489+0000 7fad06aea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:16:02.840 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.577+0000 7fad06aea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:16:02.840 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.675+0000 7fad06aea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:16:02.840 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:02 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:02.853+0000 7fad06aea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:16:02.853 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-03-11T21:16:02.854 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 9... 2024-03-11T21:16:04.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:03 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:03.860+0000 7fad06aea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:16:04.207 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:03 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:03.995+0000 7fad06aea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:16:04.613 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:04 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:04.333+0000 7fad06aea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:16:04.613 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:04 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:04.426+0000 7fad06aea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:16:04.613 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:04 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:04.626+0000 7fad06aea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:16:04.957 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:04 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:04.816+0000 7fad06aea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:16:05.357 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:05.108+0000 7fad06aea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:16:05.357 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:05.370+0000 7fad06aea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:16:05.707 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:05.614+0000 7fad06aea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:16:06.707 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:06.448+0000 7fad06aea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:16:07.345 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:07.034+0000 7fad06aea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:16:07.345 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:07.135+0000 7fad06aea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:16:07.345 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:16:07.358+0000 7fad06aea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: Active manager daemon y restarted 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: Activating manager daemon y 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: mgrmap e10: y(active, starting, since 0.00304846s) 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:16:07.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:16:07.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:16:07.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:16:07.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: Manager daemon y is now available 2024-03-11T21:16:07.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:07.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:08.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:16:08.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:16:08.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:08 smithi001 ceph-mon[25774]: [11/Mar/2024:21:16:08] ENGINE Bus STARTING 2024-03-11T21:16:08.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:08 smithi001 ceph-mon[25774]: mgrmap e11: y(active, since 1.00602s) 2024-03-11T21:16:09.387 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:16:09.387 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T21:16:09.387 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:16:09.387 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:16:09.387 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 9 is available 2024-03-11T21:16:09.387 INFO:teuthology.orchestra.run.smithi001.stdout:Generating a dashboard self-signed certificate... 2024-03-11T21:16:10.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:10 smithi001 ceph-mon[25774]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:16:10.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:10 smithi001 ceph-mon[25774]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:16:10.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:10 smithi001 ceph-mon[25774]: [11/Mar/2024:21:16:08] ENGINE Serving on https://172.21.15.1:7150 2024-03-11T21:16:10.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:10 smithi001 ceph-mon[25774]: [11/Mar/2024:21:16:08] ENGINE Serving on http://172.21.15.1:8765 2024-03-11T21:16:10.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:10 smithi001 ceph-mon[25774]: [11/Mar/2024:21:16:08] ENGINE Bus STARTED 2024-03-11T21:16:10.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:10 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:10.716 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T21:16:10.716 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial admin user... 2024-03-11T21:16:11.166 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:11 smithi001 ceph-mon[25774]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:11.167 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:11 smithi001 ceph-mon[25774]: mgrmap e12: y(active, since 2s) 2024-03-11T21:16:11.167 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:11.167 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:11.167 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:12.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:12 smithi001 ceph-mon[25774]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:12.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:12 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:12.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$/64sFhokqbCeu0GVZ624Penq6nJcQ2erWphqXrp104StKWBPvt.FG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710191771, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T21:16:12.879 INFO:teuthology.orchestra.run.smithi001.stdout:Fetching dashboard port number... 2024-03-11T21:16:13.864 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:13 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2353075982' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T21:16:14.167 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T21:16:14.177 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:16:14.177 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-03-11T21:16:14.184 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:16:14.184 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-03-11T21:16:14.184 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-03-11T21:16:14.184 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T21:16:14.186 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph Dashboard is now available at: 2024-03-11T21:16:14.186 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:14.186 INFO:teuthology.orchestra.run.smithi001.stdout: URL: https://smithi001.front.sepia.ceph.com:8443/ 2024-03-11T21:16:14.186 INFO:teuthology.orchestra.run.smithi001.stdout: User: admin 2024-03-11T21:16:14.186 INFO:teuthology.orchestra.run.smithi001.stdout: Password: x9tm65uu8b 2024-03-11T21:16:14.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:14.187 INFO:teuthology.orchestra.run.smithi001.stdout:Saving cluster configuration to /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config directory 2024-03-11T21:16:14.187 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling autotune for osd_memory_target 2024-03-11T21:16:14.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:14 smithi001 ceph-mon[25774]: mgrmap e13: y(active, since 6s) 2024-03-11T21:16:16.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T21:16:16.879 INFO:teuthology.orchestra.run.smithi001.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T21:16:16.879 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.879 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:16.879 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.879 INFO:teuthology.orchestra.run.smithi001.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: ceph telemetry on 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout:For more information see: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:16.880 INFO:teuthology.orchestra.run.smithi001.stdout:Bootstrap complete. 2024-03-11T21:16:16.921 INFO:tasks.cephadm:Fetching config... 2024-03-11T21:16:16.921 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:16:16.921 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T21:16:16.939 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T21:16:16.939 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:16:16.939 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T21:16:16.999 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T21:16:16.999 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:16:16.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T21:16:17.068 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T21:16:17.068 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:16:17.068 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T21:16:17.126 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T21:16:17.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:16:17.206 INFO:teuthology.orchestra.run.smithi001.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:16:17.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:16:17.266 INFO:teuthology.orchestra.run.smithi017.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:16:17.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:16:17.327 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6pA0NReQP/OSjKoWdLchaIT0agyfWb4QmeVErTSXKRz+JBDieewWlPkg9LMXsipdJLTgNrIzEnmuZyJvSB1wAu/RiNasBHysH7MLgmv3bJAfygivyc9B3lNfnGEv0VKDBy+qgIJeiy9qj89lL1m/HueIWbZBujgYeLnhvFVFV7sxJUHLptA/u8MzbNNH8nLHY/QM62iANMtQ7XzK7QpAorbuBVRoZ7FqXVsuIgFQlUDx3e380xT/EGdXcc3x6ev/vQBDpOcHa6CxG1iA/xX+oBHpFnYurJ0caO6Z0yBr9ERSjT6NJ6SyRz3JwGO8t4zuKIUxVD+bouS1nx27/9mdLHt/IZZP2re6PKO9jFyokxy+L49OhtbTIg0MSy2Pv5zBvPKXVKF0kFOnd873H0Wfd9GfK0u9M8ahfOUdPvvUr3QdhlglYSLl9/Um/mKW5wcz2bgvabPjZ/7DrFYX0sme265+O3G50Dv2qPWs6VbsXw/z93DBEQs5Kb8xgbRl2ovk= ceph-415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:16:17.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T21:16:17.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:17 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/3925741741' entity='client.admin' 2024-03-11T21:16:17.583 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:20.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:19 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:20.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:19 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:20.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:19 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:20.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:19 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:20.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:19 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:20.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:19 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:21.083 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T21:16:21.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T21:16:21.635 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:21 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1753981845' entity='client.admin' 2024-03-11T21:16:21.636 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:21 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:21.636 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:21 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:21.732 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:24.187 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi017 2024-03-11T21:16:24.187 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:16:24.187 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:16:24.204 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:16:24.205 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:24.262 INFO:tasks.cephadm:Adding host smithi017 to orchestrator... 2024-03-11T21:16:24.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch host add smithi017 2024-03-11T21:16:24.509 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:24.758 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:24.758 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: Updating smithi001:/etc/ceph/ceph.conf 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:24.759 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:26.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:26 smithi001 ceph-mon[25774]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:27.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:27 smithi001 ceph-mon[25774]: Deploying cephadm binary to smithi017 2024-03-11T21:16:28.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:28 smithi001 ceph-mon[25774]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:30.700 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi017' with addr '172.21.15.17' 2024-03-11T21:16:30.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:30 smithi001 ceph-mon[25774]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:31.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:16:31.560 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:31.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:31.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:31 smithi001 ceph-mon[25774]: Added host smithi017 2024-03-11T21:16:31.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:31.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:33.106 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:32 smithi001 ceph-mon[25774]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:33.106 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:32 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:33.549 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:33.549 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}] 2024-03-11T21:16:34.193 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-03-11T21:16:34.193 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:16:34.194 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:16:34.213 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:16:34.213 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:34.271 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-03-11T21:16:34.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch host add smithi204 2024-03-11T21:16:34.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:34 smithi001 ceph-mon[25774]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:34.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:34 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:34.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:34 smithi001 ceph-mon[25774]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:34.512 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:36.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:36 smithi001 ceph-mon[25774]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:36.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:36 smithi001 ceph-mon[25774]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:16:37.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:37.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: Deploying cephadm binary to smithi204 2024-03-11T21:16:37.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:16:37.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:37.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:37.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:38.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:38 smithi001 ceph-mon[25774]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:40.792 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-03-11T21:16:40.808 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:40 smithi001 ceph-mon[25774]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:41.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:16:41.693 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:42.094 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:41 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:42.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:41 smithi001 ceph-mon[25774]: Added host smithi204 2024-03-11T21:16:42.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:41 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:42.095 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:41 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:43.105 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:42 smithi001 ceph-mon[25774]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:43.105 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:42 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:43.592 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:16:43.593 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-03-11T21:16:44.130 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T21:16:44.130 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T21:16:44.366 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:16:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:44 smithi001 ceph-mon[25774]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:44 smithi001 ceph-mon[25774]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:46.392 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-03-11T21:16:46.681 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:46 smithi001 ceph-mon[25774]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:46.681 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:46 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1017891219' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:16:46.988 INFO:tasks.cephadm:Adding mon.a on smithi001 2024-03-11T21:16:46.988 INFO:tasks.cephadm:Adding mon.b on smithi017 2024-03-11T21:16:46.988 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-03-11T21:16:46.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c' 2024-03-11T21:16:47.225 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T21:16:47.273 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1017891219' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:47.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:47.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-11T21:16:47.708 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:47 smithi001 ceph-mon[25774]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:16:49.011 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-03-11T21:16:49.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:48 smithi001 ceph-mon[25774]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:49.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:48 smithi001 ceph-mon[25774]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:49.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:48 smithi001 ceph-mon[25774]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:16:49.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:49.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:49.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:49.591 DEBUG:teuthology.orchestra.run.smithi017:mon.b> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T21:16:49.594 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T21:16:49.596 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:16:49.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:16:50.289 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: Saving service mon spec with placement smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c;count:3 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:50.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:16:50.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:50.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:50 smithi001 ceph-mon[25774]: Deploying daemon mon.c on smithi204 2024-03-11T21:16:51.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:51 smithi001 ceph-mon[25774]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:52.961 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:16:52.962 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"415704e2-dfec-11ee-95c8-87774f69a715","modified":"2024-03-11T21:15:01.394648Z","created":"2024-03-11T21:15:01.394648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T21:16:52.962 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-03-11T21:16:53.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:53 smithi001 ceph-mon[25774]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:53.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:53 smithi001 ceph-mon[25774]: from='client.? 172.21.15.204:0/2658126918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:16:54.176 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 podman[30253]: 2024-03-11 21:16:54.190922156 +0000 UTC m=+0.133259901 container create c22009e5b03a0de2b12fb288c83c9c15766e04978cd6479718d44bc2dd172923 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-11T21:16:54.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 podman[30253]: 2024-03-11 21:16:54.401489678 +0000 UTC m=+0.343827378 container init c22009e5b03a0de2b12fb288c83c9c15766e04978cd6479718d44bc2dd172923 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T21:16:54.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 podman[30253]: 2024-03-11 21:16:54.406983327 +0000 UTC m=+0.349321030 container start c22009e5b03a0de2b12fb288c83c9c15766e04978cd6479718d44bc2dd172923 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:16:54.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:16:54.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T21:16:54.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pidfile_write: ignore empty --pid-file 2024-03-11T21:16:54.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: load: jerasure load: lrc 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Git sha 0 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: DB SUMMARY 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: DB Session ID: IMA44AAQ6N95KES1D36F 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:16:54.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.env: 0x559dc7e21b20 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.info_log: 0x559dc8e498e0 2024-03-11T21:16:54.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:16:54.436 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.statistics: (nil) 2024-03-11T21:16:54.436 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.use_fsync: 0 2024-03-11T21:16:54.436 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.db_log_dir: 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.wal_dir: 2024-03-11T21:16:54.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.write_buffer_manager: 0x559dc8e1d400 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:16:54.438 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.unordered_write: 0 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.row_cache: None 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.wal_filter: None 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:16:54.439 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.wal_compression: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:16:54.440 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_open_files: -1 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:16:54.441 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Compression algorithms supported: 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kZSTD supported: 0 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:16:54.442 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.merge_operator: 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_filter: None 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559dc8e499e0) 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:16:54.443 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_type: 0 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_index_type: 0 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_shortening: 1 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: checksum: 4 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: no_block_cache: 0 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache: 0x559dc8e4f090 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_options: 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: capacity : 536870912 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_shard_bits : 4 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: strict_capacity_limit : 0 2024-03-11T21:16:54.444 INFO:journalctl@ceph.mon.c.smithi204.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_compressed: (nil) 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: persistent_cache: (nil) 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size: 4096 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size_deviation: 10 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_restart_interval: 16 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_block_restart_interval: 1 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: metadata_block_size: 4096 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: partition_filters: 0 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: use_delta_encoding: 1 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: filter_policy: bloomfilter 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: whole_key_filtering: 1 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: verify_compression: 0 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:16:54.445 INFO:journalctl@ceph.mon.c.smithi204.stdout: format_version: 5 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_index_compression: 1 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_align: 0 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout: max_auto_readahead_size: 262144 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout: prepopulate_block_cache: 0 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression: NoCompression 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.num_levels: 7 2024-03-11T21:16:54.446 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:16:54.447 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:16:54.448 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:16:54.449 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:16:54.450 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.ttl: 2592000 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:16:54.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f185b2d0-3ad4-47d2-8eb0-9232c0ade74a 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191814447994, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191814448565, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710191814, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f185b2d0-3ad4-47d2-8eb0-9232c0ade74a", "db_session_id": "IMA44AAQ6N95KES1D36F", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191814448674, "job": 1, "event": "recovery_finished"} 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:16:54.452 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559dc8f42000 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: DB pointer 0x559dc8f2c000 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** DB Stats ** 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-03-11T21:16:54.453 INFO:journalctl@ceph.mon.c.smithi204.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:16:54.454 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache BinnedLRUCache@0x559dc8e4f090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: using public_addr v2:172.21.15.204:0/0 -> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-03-11T21:16:54.455 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: starting mon.c rank -1 at public addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] at bind addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:16:54.643 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:16:54.644 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:16:54.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(???) e0 preinit fsid 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:16:54.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T21:16:54.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T21:16:54.803 INFO:journalctl@ceph.mon.c.smithi204.stdout: e1 2024-03-11T21:16:54.803 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout: legacy client fscid: -1 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout: No filesystems configured 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:16:54.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:54.805 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Deploying cephadm binary to smithi204 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.806 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Added host smithi204 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.807 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1017891219' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:16:54.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1017891219' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:16:54.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:16:54.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:16:54.810 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Saving service mon spec with placement smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c;count:3 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:54.811 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: Deploying daemon mon.c on smithi204 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: from='client.? 172.21.15.204:0/2658126918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 ceph-mon[30291]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 bash[30253]: c22009e5b03a0de2b12fb288c83c9c15766e04978cd6479718d44bc2dd172923 2024-03-11T21:16:54.812 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:54 smithi204 systemd[1]: Started Ceph mon.c for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:16:54.877 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:16:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:16:59.822 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"415704e2-dfec-11ee-95c8-87774f69a715","modified":"2024-03-11T21:16:54.472682Z","created":"2024-03-11T21:15:01.394648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:16:59.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:16:59.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: mon.a calling monitor election 2024-03-11T21:16:59.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:59.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.822 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: mon.c calling monitor election 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: monmap e2: 2 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: fsmap 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: mgrmap e13: y(active, since 52s) 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: overall HEALTH_OK 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:59.823 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:59.824 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:59.824 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:16:59.824 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:16:59 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:59.824 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-03-11T21:16:59.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:16:59.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: mon.a calling monitor election 2024-03-11T21:16:59.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:59.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: mon.c calling monitor election 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: monmap e2: 2 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: fsmap 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: mgrmap e13: y(active, since 52s) 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: overall HEALTH_OK 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:16:59.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:16:59.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:16:59 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:00.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:00 smithi204 ceph-mon[30291]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:00.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:00 smithi204 ceph-mon[30291]: Deploying daemon mon.b on smithi017 2024-03-11T21:17:00.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:00 smithi204 ceph-mon[30291]: from='client.? 172.21.15.204:0/2776825260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:00.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:00 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:00.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:00 smithi001 ceph-mon[25774]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:00.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:00 smithi001 ceph-mon[25774]: Deploying daemon mon.b on smithi017 2024-03-11T21:17:00.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:00 smithi001 ceph-mon[25774]: from='client.? 172.21.15.204:0/2776825260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:00.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:00 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:01.492 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:17:01.492 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:17:01.739 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:17:02.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:02 smithi204 ceph-mon[30291]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:02.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:02 smithi001 ceph-mon[25774]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 podman[29824]: 2024-03-11 21:17:03.082590762 +0000 UTC m=+0.017659346 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:17:03.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 podman[29824]: 2024-03-11 21:17:03.358814131 +0000 UTC m=+0.293882716 container init 28dccf6c3addd7b0cb8410f8aadb19e8701dab8e334849551877b3f0ed3e8403 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T21:17:03.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 podman[29824]: 2024-03-11 21:17:03.364080213 +0000 UTC m=+0.299148788 container start 28dccf6c3addd7b0cb8410f8aadb19e8701dab8e334849551877b3f0ed3e8403 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-11T21:17:03.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:17:03.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T21:17:03.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pidfile_write: ignore empty --pid-file 2024-03-11T21:17:03.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: load: jerasure load: lrc 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Git sha 0 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: DB SUMMARY 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: DB Session ID: VVCD0HXHQYKFS7WXAW9L 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:17:03.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.env: 0x556fb0f00b20 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.info_log: 0x556fb30dd8e0 2024-03-11T21:17:03.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.statistics: (nil) 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.use_fsync: 0 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:17:03.680 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.db_log_dir: 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.wal_dir: 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:17:03.681 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.write_buffer_manager: 0x556fb30b1400 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:17:03.682 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.unordered_write: 0 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.row_cache: None 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.wal_filter: None 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:17:03.683 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.wal_compression: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:17:03.684 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:17:03.685 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_open_files: -1 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Compression algorithms supported: 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kZSTD supported: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:17:03.686 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.merge_operator: 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_filter: None 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:17:03.687 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556fb30dd9e0) 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: index_type: 0 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: data_block_index_type: 0 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: index_shortening: 1 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:17:03.688 INFO:journalctl@ceph.mon.b.smithi017.stdout: checksum: 4 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: no_block_cache: 0 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_cache: 0x556fb30e3090 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_cache_options: 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: capacity : 536870912 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: num_shard_bits : 4 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: strict_capacity_limit : 0 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_cache_compressed: (nil) 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: persistent_cache: (nil) 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_size: 4096 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_size_deviation: 10 2024-03-11T21:17:03.689 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_restart_interval: 16 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: index_block_restart_interval: 1 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: metadata_block_size: 4096 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: partition_filters: 0 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: use_delta_encoding: 1 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: filter_policy: bloomfilter 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: whole_key_filtering: 1 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: verify_compression: 0 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: format_version: 5 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: enable_index_compression: 1 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: block_align: 0 2024-03-11T21:17:03.690 INFO:journalctl@ceph.mon.b.smithi017.stdout: max_auto_readahead_size: 262144 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout: prepopulate_block_cache: 0 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression: NoCompression 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.num_levels: 7 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:17:03.691 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:17:03.692 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:17:03.693 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:17:03.694 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:17:03.695 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:17:03.696 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.ttl: 2592000 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:17:03.697 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 694ca5b7-7e1c-46da-89a5-4a297a17afc3 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191823394856, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191823395352, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710191823, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "694ca5b7-7e1c-46da-89a5-4a297a17afc3", "db_session_id": "VVCD0HXHQYKFS7WXAW9L", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:17:03.698 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710191823395436, "job": 1, "event": "recovery_finished"} 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556fb31d6000 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: DB pointer 0x556fb31c0000 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout: ** DB Stats ** 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:17:03.699 INFO:journalctl@ceph.mon.b.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:17:03.700 INFO:journalctl@ceph.mon.b.smithi017.stdout: 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:17:03.701 INFO:journalctl@ceph.mon.b.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: Block cache BinnedLRUCache@0x556fb30e3090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: using public_addr v2:172.21.15.17:0/0 -> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] 2024-03-11T21:17:03.702 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: starting mon.b rank -1 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(???) e0 preinit fsid 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout: e1 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout: legacy client fscid: -1 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout: 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout: No filesystems configured 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:17:03.703 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.704 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-11T21:17:03.705 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Deploying cephadm binary to smithi204 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Added host smithi204 2024-03-11T21:17:03.706 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1017891219' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1017891219' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.707 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.708 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Saving service mon spec with placement smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c;count:3 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:03.709 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Deploying daemon mon.c on smithi204 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.? 172.21.15.204:0/2658126918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.a calling monitor election 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.c calling monitor election 2024-03-11T21:17:03.710 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: monmap e2: 2 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: fsmap 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mgrmap e13: y(active, since 52s) 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: overall HEALTH_OK 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.711 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: Deploying daemon mon.b on smithi017 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='client.? 172.21.15.204:0/2776825260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 ceph-mon[29862]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 bash[29824]: 28dccf6c3addd7b0cb8410f8aadb19e8701dab8e334849551877b3f0ed3e8403 2024-03-11T21:17:03.712 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:03 smithi017 systemd[1]: Started Ceph mon.b for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:17:08.437 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-11T21:17:08.438 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"415704e2-dfec-11ee-95c8-87774f69a715","modified":"2024-03-11T21:17:03.418384Z","created":"2024-03-11T21:15:01.394648Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:17:08.440 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-03-11T21:17:08.778 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: mon.c calling monitor election 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: mon.a calling monitor election 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='client.? 172.21.15.204:0/2544838472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: monmap e3: 3 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],b=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: fsmap 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: mgrmap e13: y(active, since 61s) 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: mon.b (rank 2) addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] is down (out of quorum) 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.780 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: mon.c calling monitor election 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: mon.a calling monitor election 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='client.? 172.21.15.204:0/2544838472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.958 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: monmap e3: 3 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],b=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: fsmap 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: mgrmap e13: y(active, since 61s) 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: mon.b (rank 2) addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] is down (out of quorum) 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.959 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.960 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:08.960 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:09.022 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T21:17:09.022 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T21:17:09.273 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:09.643 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:09 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:09.643 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:09 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:09.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:09 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:09.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:09 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:10.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:17:10.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: mon.c calling monitor election 2024-03-11T21:17:10.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: mon.a calling monitor election 2024-03-11T21:17:10.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='client.? 172.21.15.204:0/2544838472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: monmap e3: 3 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],b=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: fsmap 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: mgrmap e13: y(active, since 61s) 2024-03-11T21:17:10.677 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: mon.b (rank 2) addr [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] is down (out of quorum) 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:10.678 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:11.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: mon.b calling monitor election 2024-03-11T21:17:11.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: mon.b calling monitor election 2024-03-11T21:17:11.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: mon.c calling monitor election 2024-03-11T21:17:11.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: mon.a calling monitor election 2024-03-11T21:17:11.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:17:11.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: monmap e3: 3 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],b=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: fsmap 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: mgrmap e13: y(active, since 63s) 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: Cluster is now healthy 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: overall HEALTH_OK 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.208 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.209 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.209 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.209 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:11.209 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:11.209 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:11.253 INFO:teuthology.orchestra.run.smithi001.stdout:# minimal ceph.conf for 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:17:11.253 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-03-11T21:17:11.253 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:17:11.254 INFO:teuthology.orchestra.run.smithi001.stdout: mon_host = [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: mon.b calling monitor election 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: mon.b calling monitor election 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: mon.c calling monitor election 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: mon.a calling monitor election 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: monmap e3: 3 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],b=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: fsmap 2024-03-11T21:17:11.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: mgrmap e13: y(active, since 63s) 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: Cluster is now healthy 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: overall HEALTH_OK 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.304 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.305 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.305 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:11.305 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:11.305 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:11.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: mon.b calling monitor election 2024-03-11T21:17:11.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: mon.b calling monitor election 2024-03-11T21:17:11.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: mon.c calling monitor election 2024-03-11T21:17:11.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: mon.a calling monitor election 2024-03-11T21:17:11.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: monmap e3: 3 mons at {a=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],b=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: fsmap 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: mgrmap e13: y(active, since 63s) 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: Cluster is now healthy 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: overall HEALTH_OK 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.427 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:11.428 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:12.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:12 smithi204 ceph-mon[30291]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:17:12.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:12 smithi204 ceph-mon[30291]: Reconfiguring daemon mon.a on smithi001 2024-03-11T21:17:12.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:12 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1227284070' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:12.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:12 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:12.375 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:12 smithi001 ceph-mon[25774]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:17:12.375 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:12 smithi001 ceph-mon[25774]: Reconfiguring daemon mon.a on smithi001 2024-03-11T21:17:12.375 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:12 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1227284070' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:12.376 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:12 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:12.400 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T21:17:12.400 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:17:12.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:17:12.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:12 smithi017 ceph-mon[29862]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:17:12.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:12 smithi017 ceph-mon[29862]: Reconfiguring daemon mon.a on smithi001 2024-03-11T21:17:12.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:12 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1227284070' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:12.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:12 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:17:12.430 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:17:12.430 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:17:12.499 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:17:12.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:17:12.533 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:17:12.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:17:12.604 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:17:12.604 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:17:12.638 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:17:12.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:17:12.709 INFO:tasks.cephadm:Adding mgr.y on smithi001 2024-03-11T21:17:12.709 INFO:tasks.cephadm:Adding mgr.x on smithi017 2024-03-11T21:17:12.710 INFO:tasks.cephadm:Adding mgr.z on smithi204 2024-03-11T21:17:12.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi001=y;smithi017=x;smithi204=z' 2024-03-11T21:17:12.986 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:17:13.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:13.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:13.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:13.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:13 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:13.259 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:13.260 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.260 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.260 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.260 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:13.260 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:13.260 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:13 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:13.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:13.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:13.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:13.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:13.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:13 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:14.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:14 smithi204 ceph-mon[30291]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:17:14.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:14 smithi204 ceph-mon[30291]: Reconfiguring daemon mon.b on smithi017 2024-03-11T21:17:14.328 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:14 smithi017 ceph-mon[29862]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:17:14.328 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:14 smithi017 ceph-mon[29862]: Reconfiguring daemon mon.b on smithi017 2024-03-11T21:17:14.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:14 smithi001 ceph-mon[25774]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:17:14.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:14 smithi001 ceph-mon[25774]: Reconfiguring daemon mon.b on smithi017 2024-03-11T21:17:14.722 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:15.261 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.262 DEBUG:teuthology.orchestra.run.smithi017:mgr.x> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T21:17:15.265 DEBUG:teuthology.orchestra.run.smithi204:mgr.z> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T21:17:15.267 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T21:17:15.267 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:17:15.267 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:17:15.286 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:17:15.286 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/nvme1n1 2024-03-11T21:17:15.290 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:15.291 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.291 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.291 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:15.291 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:15.291 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:15.291 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/nvme1n1 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:16:20.479547071 +0000 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:45.666235199 +0000 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:45.666235199 +0000 2024-03-11T21:17:15.345 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:45.659234907 +0000 2024-03-11T21:17:15.346 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:17:15.411 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:15.411 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.411 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.411 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:17:15.412 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:17:15.412 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:15.412 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:15.414 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:17:15.414 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:17:15.414 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000146801 s, 3.5 MB/s 2024-03-11T21:17:15.415 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:17:15.475 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/nvme2n1 2024-03-11T21:17:15.530 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/nvme2n1 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:16:20.488546970 +0000 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:45.819241568 +0000 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:45.819241568 +0000 2024-03-11T21:17:15.531 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:45.808241110 +0000 2024-03-11T21:17:15.531 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:17:15.597 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:17:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:17:15.598 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000135008 s, 3.8 MB/s 2024-03-11T21:17:15.599 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:17:15.656 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/nvme3n1 2024-03-11T21:17:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/nvme3n1 2024-03-11T21:17:15.712 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:15.712 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-11T21:17:15.713 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:15.713 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:15.713 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:16:20.498546857 +0000 2024-03-11T21:17:15.713 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:45.965247646 +0000 2024-03-11T21:17:15.713 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:45.965247646 +0000 2024-03-11T21:17:15.713 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:45.955247230 +0000 2024-03-11T21:17:15.713 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:17:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:17:15.779 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:17:15.780 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000117396 s, 4.4 MB/s 2024-03-11T21:17:15.781 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:17:15.837 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/nvme4n1 2024-03-11T21:17:15.891 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/nvme4n1 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-03-11 21:16:20.508546745 +0000 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-03-11 21:13:46.114253850 +0000 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-03-11 21:13:46.114253850 +0000 2024-03-11T21:17:15.892 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-03-11 21:13:46.104253433 +0000 2024-03-11T21:17:15.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:17:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-03-11T21:17:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-03-11T21:17:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000130681 s, 3.9 MB/s 2024-03-11T21:17:15.959 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:17:16.016 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:17:16.016 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:17:16.034 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:17:16.037 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/nvme1n1 2024-03-11T21:17:16.092 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/nvme1n1 2024-03-11T21:17:16.092 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:16.092 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:17:16.093 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:16.093 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:16.093 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:16:35.577900606 +0000 2024-03-11T21:17:16.093 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:47.295742432 +0000 2024-03-11T21:17:16.093 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:47.295742432 +0000 2024-03-11T21:17:16.093 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:47.284742556 +0000 2024-03-11T21:17:16.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:17:16.162 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:17:16.162 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:17:16.162 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000131068 s, 3.9 MB/s 2024-03-11T21:17:16.164 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:17:16.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:16 smithi001 ceph-mon[25774]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:17:16.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:16 smithi001 ceph-mon[25774]: Reconfiguring daemon mon.c on smithi204 2024-03-11T21:17:16.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:16 smithi001 ceph-mon[25774]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=y;smithi017=x;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:16.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:16 smithi001 ceph-mon[25774]: Saving service mgr spec with placement smithi001=y;smithi017=x;smithi204=z;count:3 2024-03-11T21:17:16.221 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/nvme2n1 2024-03-11T21:17:16.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:16 smithi204 ceph-mon[30291]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:17:16.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:16 smithi204 ceph-mon[30291]: Reconfiguring daemon mon.c on smithi204 2024-03-11T21:17:16.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:16 smithi204 ceph-mon[30291]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=y;smithi017=x;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:16.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:16 smithi204 ceph-mon[30291]: Saving service mgr spec with placement smithi001=y;smithi017=x;smithi204=z;count:3 2024-03-11T21:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/nvme2n1 2024-03-11T21:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:16.278 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T21:17:16.279 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:16.279 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:16.279 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:16:35.585900523 +0000 2024-03-11T21:17:16.279 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:47.442740773 +0000 2024-03-11T21:17:16.279 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:47.442740773 +0000 2024-03-11T21:17:16.279 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:47.435740852 +0000 2024-03-11T21:17:16.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:17:16.346 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:16 smithi017 ceph-mon[29862]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:17:16.346 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:16 smithi017 ceph-mon[29862]: Reconfiguring daemon mon.c on smithi204 2024-03-11T21:17:16.346 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:16 smithi017 ceph-mon[29862]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=y;smithi017=x;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:16.346 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:16 smithi017 ceph-mon[29862]: Saving service mgr spec with placement smithi001=y;smithi017=x;smithi204=z;count:3 2024-03-11T21:17:16.348 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:17:16.348 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:17:16.348 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000127667 s, 4.0 MB/s 2024-03-11T21:17:16.349 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:17:16.406 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/nvme3n1 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/nvme3n1 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:16:35.594900429 +0000 2024-03-11T21:17:16.462 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:47.589739142 +0000 2024-03-11T21:17:16.463 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:47.589739142 +0000 2024-03-11T21:17:16.463 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:47.579739253 +0000 2024-03-11T21:17:16.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:17:16.528 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:17:16.528 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:17:16.528 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000130445 s, 3.9 MB/s 2024-03-11T21:17:16.529 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:17:16.585 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/nvme4n1 2024-03-11T21:17:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/nvme4n1 2024-03-11T21:17:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-11 21:16:35.602900346 +0000 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-11 21:13:47.735737521 +0000 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-11 21:13:47.735737521 +0000 2024-03-11T21:17:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-11 21:13:47.725737632 +0000 2024-03-11T21:17:16.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:17:16.708 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-11T21:17:16.708 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-11T21:17:16.708 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000120422 s, 4.3 MB/s 2024-03-11T21:17:16.709 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:17:16.765 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:17:16.765 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:17:16.782 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:17:16.782 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme1n1 2024-03-11T21:17:16.838 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme1n1 2024-03-11T21:17:16.838 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:16:45.970946546 +0000 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:48.950661953 +0000 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:48.950661953 +0000 2024-03-11T21:17:16.839 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:48.938662137 +0000 2024-03-11T21:17:16.839 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:17:16.906 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:17:16.907 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:17:16.907 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000276551 s, 1.9 MB/s 2024-03-11T21:17:16.908 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:17:16.964 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme2n1 2024-03-11T21:17:17.018 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme2n1 2024-03-11T21:17:17.018 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:17.018 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T21:17:17.018 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:17.018 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:17.019 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:16:45.981946379 +0000 2024-03-11T21:17:17.019 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:49.097659693 +0000 2024-03-11T21:17:17.019 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:49.097659693 +0000 2024-03-11T21:17:17.019 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:49.087659847 +0000 2024-03-11T21:17:17.019 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:17:17.085 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:17:17.085 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:17:17.085 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00019827 s, 2.6 MB/s 2024-03-11T21:17:17.086 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:17:17.142 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme3n1 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme3n1 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:16:45.991946227 +0000 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:49.245657418 +0000 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:49.245657418 +0000 2024-03-11T21:17:17.198 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:49.240657495 +0000 2024-03-11T21:17:17.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:17:17.264 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:17:17.264 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:17:17.264 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000232899 s, 2.2 MB/s 2024-03-11T21:17:17.266 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:17:17.321 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/nvme4n1 2024-03-11T21:17:17.378 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/nvme4n1 2024-03-11T21:17:17.378 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-11 21:16:46.002946060 +0000 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-11 21:13:49.401655021 +0000 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-11 21:13:49.401655021 +0000 2024-03-11T21:17:17.379 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-11 21:13:49.390655190 +0000 2024-03-11T21:17:17.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:17:17.448 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-11T21:17:17.448 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-11T21:17:17.448 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000239314 s, 2.1 MB/s 2024-03-11T21:17:17.449 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:17:17.505 INFO:tasks.cephadm:Deploying osd.0 on smithi001 with /dev/nvme4n1... 2024-03-11T21:17:17.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:17:17.513 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:17.514 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:17 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:17.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:17:17.534 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:17.534 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:17 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:17.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:17.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:17 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:18.531 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:18 smithi204 ceph-mon[30291]: Deploying daemon mgr.z on smithi204 2024-03-11T21:17:18.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:18 smithi017 ceph-mon[29862]: Deploying daemon mgr.z on smithi204 2024-03-11T21:17:18.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:18 smithi001 ceph-mon[25774]: Deploying daemon mgr.z on smithi204 2024-03-11T21:17:19.193 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:18 smithi204 systemd[1]: Starting Ceph mgr.z for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:17:19.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:19 smithi001 ceph-mon[25774]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:19 smithi204 ceph-mon[30291]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 podman[32355]: 2024-03-11 21:17:19.111553615 +0000 UTC m=+0.018300942 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 podman[32355]: 2024-03-11 21:17:19.21542973 +0000 UTC m=+0.122177043 container create 54b95e8edd1a4de9b3634a16fb297498df4ee78b087ae9cb2bf92be132b653e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 podman[32355]: 2024-03-11 21:17:19.409449319 +0000 UTC m=+0.316196638 container init 54b95e8edd1a4de9b3634a16fb297498df4ee78b087ae9cb2bf92be132b653e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 podman[32355]: 2024-03-11 21:17:19.415224277 +0000 UTC m=+0.321971592 container start 54b95e8edd1a4de9b3634a16fb297498df4ee78b087ae9cb2bf92be132b653e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS) 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 ceph-mgr[32394]: -- 172.21.15.204:0/1663639435 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a5caae25a0 con 0x55a5caab7000 2024-03-11T21:17:19.511 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 bash[32355]: 54b95e8edd1a4de9b3634a16fb297498df4ee78b087ae9cb2bf92be132b653e1 2024-03-11T21:17:19.577 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:19 smithi017 ceph-mon[29862]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:19.803 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 systemd[1]: Started Ceph mgr.z for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00948749 s, 1.1 GB/s 2024-03-11T21:17:20.079 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-03-11T21:17:20.302 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:19 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:19.959+0000 7f342efe7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:17:20.302 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:20.052+0000 7f342efe7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:17:20.690 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:20.690 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.690 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.690 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.690 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:20.690 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:17:20.691 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:20.691 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:20.691 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:20 smithi001 ceph-mon[25774]: Deploying daemon mgr.x on smithi017 2024-03-11T21:17:20.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi001:/dev/nvme4n1 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:20.691+0000 7f342efe7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:20.784+0000 7f342efe7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:20.924+0000 7f342efe7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.910 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:20.911 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:17:20.911 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:20.911 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:20.911 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:20 smithi204 ceph-mon[30291]: Deploying daemon mgr.x on smithi017 2024-03-11T21:17:20.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:20.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:20 smithi017 ceph-mon[29862]: Deploying daemon mgr.x on smithi017 2024-03-11T21:17:20.998 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:17:21.302 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.018+0000 7f342efe7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:17:21.303 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.099+0000 7f342efe7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:17:21.629 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.359+0000 7f342efe7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:17:21.630 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.466+0000 7f342efe7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:17:21.630 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.551+0000 7f342efe7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:17:21.630 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.643+0000 7f342efe7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:17:22.025 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:21 smithi017 podman[31299]: 2024-03-11 21:17:21.846777368 +0000 UTC m=+0.018485572 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:17:22.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:21 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:21.819+0000 7f342efe7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:17:22.426 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 podman[31299]: 2024-03-11 21:17:22.145053217 +0000 UTC m=+0.316761429 container init e616fb4a563295fdcb99bfa7f3c8b67285449e67b4123a95c41303d24bef8656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:17:22.426 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 podman[31299]: 2024-03-11 21:17:22.150442014 +0000 UTC m=+0.322150215 container start e616fb4a563295fdcb99bfa7f3c8b67285449e67b4123a95c41303d24bef8656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:17:22.426 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 ceph-mgr[31337]: -- 172.21.15.17:0/118918045 <== mon.2 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b07cc9e5a0 con 0x55b07cc73000 2024-03-11T21:17:22.426 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 bash[31299]: e616fb4a563295fdcb99bfa7f3c8b67285449e67b4123a95c41303d24bef8656 2024-03-11T21:17:22.426 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 systemd[1]: Started Ceph mgr.x for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:17:23.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:22 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:22.808+0000 7f342efe7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:17:23.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:22 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:22.944+0000 7f342efe7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:17:23.175 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:22.712+0000 7fedb8431200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:17:23.176 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:22 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:22.809+0000 7fedb8431200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:23.304+0000 7f342efe7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:23.400+0000 7f342efe7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:23.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:23.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:23.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:23.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:23.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:23.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:23.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:23.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:23.500+0000 7fedb8431200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:23.595+0000 7fedb8431200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:23.581 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:23.582 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:23.582 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:23.582 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:23.914 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:23.739+0000 7fedb8431200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:17:23.914 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:23.840+0000 7fedb8431200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:17:23.914 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:23 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:23.927+0000 7fedb8431200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:17:24.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:23.598+0000 7f342efe7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:17:24.053 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:23 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:23.793+0000 7f342efe7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:17:24.322 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:24.075+0000 7f342efe7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:17:24.322 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:24.336+0000 7f342efe7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:17:24.322 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:24.323 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:24.479 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:24.480 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.480 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.480 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.480 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.480 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:24.480 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:24.481 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.481 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:24.481 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:24.481 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:24.482 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:24.196+0000 7fedb8431200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:17:24.482 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:24.308+0000 7fedb8431200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:17:24.482 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:24.396+0000 7fedb8431200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:17:24.482 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:24.493+0000 7fedb8431200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:17:24.613 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:24.613 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.613 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:17:24.614 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:24 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:24.802 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:24 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:24.570+0000 7f342efe7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:17:24.925 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:24 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:24.670+0000 7fedb8431200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:17:25.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:25 smithi204 ceph-mon[30291]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:25.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:25 smithi204 ceph-mon[30291]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:17:25.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:25 smithi204 ceph-mon[30291]: Reconfiguring daemon mgr.y on smithi001 2024-03-11T21:17:25.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:25 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:25.364+0000 7f342efe7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:17:25.652 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:25 smithi017 ceph-mon[29862]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:25.652 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:25 smithi017 ceph-mon[29862]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:17:25.652 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:25 smithi017 ceph-mon[29862]: Reconfiguring daemon mgr.y on smithi001 2024-03-11T21:17:25.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:25 smithi001 ceph-mon[25774]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:25.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:25 smithi001 ceph-mon[25774]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:17:25.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:25 smithi001 ceph-mon[25774]: Reconfiguring daemon mgr.y on smithi001 2024-03-11T21:17:25.925 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:25 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:25.666+0000 7fedb8431200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:17:25.925 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:25 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:25.799+0000 7fedb8431200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:17:26.226 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:25 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:25.931+0000 7f342efe7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:17:26.226 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:26.028+0000 7f342efe7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:17:26.226 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:17:26.240+0000 7f342efe7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:17:26.414 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2495638395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6228c6e6-acf8-40c7-8960-b8591761b393"}]: dispatch 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2495638395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6228c6e6-acf8-40c7-8960-b8591761b393"}]': finished 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: Standby manager daemon z started 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:26.138+0000 7fedb8431200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:26.230+0000 7fedb8431200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:17:26.415 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:26.428+0000 7fedb8431200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:17:26.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2495638395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6228c6e6-acf8-40c7-8960-b8591761b393"}]: dispatch 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2495638395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6228c6e6-acf8-40c7-8960-b8591761b393"}]': finished 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: Standby manager daemon z started 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:17:26.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:26 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:17:26.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:26.616+0000 7fedb8431200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2495638395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6228c6e6-acf8-40c7-8960-b8591761b393"}]: dispatch 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2495638395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6228c6e6-acf8-40c7-8960-b8591761b393"}]': finished 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: Standby manager daemon z started 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:17:26.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:26 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/1596621494' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:17:27.156 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:26 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:26.904+0000 7fedb8431200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:17:27.156 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:27 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:27.170+0000 7fedb8431200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:17:27.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:27 smithi017 ceph-mon[29862]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:27.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:27 smithi017 ceph-mon[29862]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T21:17:27.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:27 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:17:27.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:27 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2199070797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:27.426 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:27 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:27.411+0000 7fedb8431200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:17:27.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:27 smithi204 ceph-mon[30291]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:27.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:27 smithi204 ceph-mon[30291]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T21:17:27.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:27 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:17:27.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:27 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2199070797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:27.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:27 smithi001 ceph-mon[25774]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:27.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:27 smithi001 ceph-mon[25774]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T21:17:27.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:27 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:17:27.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:27 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2199070797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:28.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:28 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:28.234+0000 7fedb8431200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:17:29.119 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:28 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:28.814+0000 7fedb8431200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:17:29.120 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:28 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:28.914+0000 7fedb8431200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:17:29.120 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:17:29.133+0000 7fedb8431200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:17:29.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-mon[29862]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:29.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-mon[29862]: Standby manager daemon x started 2024-03-11T21:17:29.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:17:29.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:17:29.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:17:29.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:29 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:17:29.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:29 smithi001 ceph-mon[25774]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:29.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:29 smithi001 ceph-mon[25774]: Standby manager daemon x started 2024-03-11T21:17:29.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:29 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:17:29.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:29 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:17:29.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:29 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:17:29.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:29 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:17:29.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:29 smithi204 ceph-mon[30291]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:29.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:29 smithi204 ceph-mon[30291]: Standby manager daemon x started 2024-03-11T21:17:29.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:29 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:17:29.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:29 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:17:29.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:29 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:17:29.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:29 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/372366217' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:17:30.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:30 smithi001 ceph-mon[25774]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T21:17:30.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:30 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:17:30.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:30 smithi017 ceph-mon[29862]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T21:17:30.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:30 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:17:30.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:30 smithi204 ceph-mon[30291]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T21:17:30.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:30 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:31.491 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:31.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:31.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:31.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:31.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:31.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:33.623 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:33 smithi001 ceph-mon[25774]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:33.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:33 smithi017 ceph-mon[29862]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:33.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:33 smithi204 ceph-mon[30291]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:34.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:34 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:17:34.563 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:34 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:34 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:17:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:34 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:34.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:34 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:17:34.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:34 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:35.664 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:35 smithi001 ceph-mon[25774]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:35.664 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:35 smithi001 ceph-mon[25774]: Deploying daemon osd.0 on smithi001 2024-03-11T21:17:35.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:35 smithi017 ceph-mon[29862]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:35.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:35 smithi017 ceph-mon[29862]: Deploying daemon osd.0 on smithi001 2024-03-11T21:17:35.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:35 smithi204 ceph-mon[30291]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:35.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:35 smithi204 ceph-mon[30291]: Deploying daemon osd.0 on smithi001 2024-03-11T21:17:37.602 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:37 smithi001 ceph-mon[25774]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:37 smithi017 ceph-mon[29862]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:37.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:37 smithi204 ceph-mon[30291]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:39.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:39 smithi017 ceph-mon[29862]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:39.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:39 smithi001 ceph-mon[25774]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:39.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:39 smithi204 ceph-mon[30291]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:40.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:40 smithi017 ceph-mon[29862]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:40.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:40.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:40 smithi001 ceph-mon[25774]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:40.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:40.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:40.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:41.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:40 smithi204 ceph-mon[30291]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:41.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:41.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:41.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:43 smithi001 ceph-mon[25774]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:17:43.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:43.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:43.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:43.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:43 smithi204 ceph-mon[30291]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:17:43.613 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 0 on host 'smithi001' 2024-03-11T21:17:43.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:43.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:43.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:43.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:43.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:43 smithi017 ceph-mon[29862]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:17:44.228 DEBUG:teuthology.orchestra.run.smithi001:osd.0> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T21:17:44.230 INFO:tasks.cephadm:Deploying osd.1 on smithi017 with /dev/nvme4n1... 2024-03-11T21:17:44.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:17:44.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:17:44.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:17:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-03-11T21:17:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:44.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:44.457 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0[44215]: 2024-03-11T21:17:44.287+0000 7f0ffa1d3700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T21:17:44.457 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:17:44 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0[44215]: 2024-03-11T21:17:44.290+0000 7f0ff5fc6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:17:44.476 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:44.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:17:44.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:17:44.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-03-11T21:17:44.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:44.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:44.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:44.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:44.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:17:44.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:17:44.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-03-11T21:17:44.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:44.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:44.598 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:44.598 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:45.328 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:45 smithi001 ceph-mon[25774]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:45.328 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:45 smithi001 ceph-mon[25774]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-03-11T21:17:45.328 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:45 smithi001 ceph-mon[25774]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:17:45.329 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:45 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:45.329 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:45 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:45.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:45 smithi204 ceph-mon[30291]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:45.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:45 smithi204 ceph-mon[30291]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-03-11T21:17:45.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:45 smithi204 ceph-mon[30291]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:17:45.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:45 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:45.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:45 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:45.631 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:45 smithi017 ceph-mon[29862]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:45.632 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:45 smithi017 ceph-mon[29862]: from='osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-03-11T21:17:45.632 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:45 smithi017 ceph-mon[29862]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:17:45.632 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:45 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:45.634 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:45 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00948092 s, 1.1 GB/s 2024-03-11T21:17:46.290 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-03-11T21:17:46.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:46 smithi204 ceph-mon[30291]: purged_snaps scrub starts 2024-03-11T21:17:46.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:46 smithi204 ceph-mon[30291]: purged_snaps scrub ok 2024-03-11T21:17:46.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:46 smithi204 ceph-mon[30291]: osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761] boot 2024-03-11T21:17:46.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:46 smithi204 ceph-mon[30291]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:17:46.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:46.670 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:46 smithi017 ceph-mon[29862]: purged_snaps scrub starts 2024-03-11T21:17:46.670 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:46 smithi017 ceph-mon[29862]: purged_snaps scrub ok 2024-03-11T21:17:46.670 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:46 smithi017 ceph-mon[29862]: osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761] boot 2024-03-11T21:17:46.670 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:46 smithi017 ceph-mon[29862]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:17:46.670 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:46.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:46 smithi001 ceph-mon[25774]: purged_snaps scrub starts 2024-03-11T21:17:46.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:46 smithi001 ceph-mon[25774]: purged_snaps scrub ok 2024-03-11T21:17:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:46 smithi001 ceph-mon[25774]: osd.0 [v2:172.21.15.1:6802/3615798761,v1:172.21.15.1:6803/3615798761] boot 2024-03-11T21:17:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:46 smithi001 ceph-mon[25774]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:17:46.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:17:46.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi017:/dev/nvme4n1 2024-03-11T21:17:47.225 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:17:47.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:47 smithi001 ceph-mon[25774]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:47.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:47 smithi001 ceph-mon[25774]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:17:47.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:47 smithi204 ceph-mon[30291]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:47.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:47 smithi204 ceph-mon[30291]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:17:47.588 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:47 smithi017 ceph-mon[29862]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:17:47.588 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:47 smithi017 ceph-mon[29862]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:17:48.645 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:48.645 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: Detected new or changed devices on smithi001 2024-03-11T21:17:48.645 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.645 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.645 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:48.646 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: Adjusting osd_memory_target on smithi001 to 17093M 2024-03-11T21:17:48.646 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.646 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:48.646 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:48.646 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: Detected new or changed devices on smithi001 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: Adjusting osd_memory_target on smithi001 to 17093M 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:48.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:49.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: Detected new or changed devices on smithi001 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: Adjusting osd_memory_target on smithi001 to 17093M 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:49.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:17:49.822 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:49 smithi017 ceph-mon[29862]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:49.822 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:49 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:49.822 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:49 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:49.822 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:49 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:49.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:49 smithi001 ceph-mon[25774]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:49.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:49 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:49.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:49 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:49.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:49 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:50.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:49 smithi204 ceph-mon[30291]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:17:50.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:49 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:17:50.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:49 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:17:50.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:49 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:50 smithi017 ceph-mon[29862]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:50.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:50 smithi001 ceph-mon[25774]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:51.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:50 smithi204 ceph-mon[30291]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:51.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:51 smithi017 ceph-mon[29862]: from='client.? 172.21.15.17:0/1534113145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]: dispatch 2024-03-11T21:17:51.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:51 smithi017 ceph-mon[29862]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]: dispatch 2024-03-11T21:17:51.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:51 smithi017 ceph-mon[29862]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]': finished 2024-03-11T21:17:51.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:51 smithi017 ceph-mon[29862]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:17:51.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:51 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:51.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:51 smithi001 ceph-mon[25774]: from='client.? 172.21.15.17:0/1534113145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]: dispatch 2024-03-11T21:17:51.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:51 smithi001 ceph-mon[25774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]: dispatch 2024-03-11T21:17:51.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:51 smithi001 ceph-mon[25774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]': finished 2024-03-11T21:17:51.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:51 smithi001 ceph-mon[25774]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:17:51.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:51 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:52.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:51 smithi204 ceph-mon[30291]: from='client.? 172.21.15.17:0/1534113145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]: dispatch 2024-03-11T21:17:52.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:51 smithi204 ceph-mon[30291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]: dispatch 2024-03-11T21:17:52.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:51 smithi204 ceph-mon[30291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b709121-8dd2-46f8-b98f-5cbf1a181d11"}]': finished 2024-03-11T21:17:52.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:51 smithi204 ceph-mon[30291]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:17:52.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:51 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:17:52.897 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:52 smithi017 ceph-mon[29862]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:52.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:52 smithi001 ceph-mon[25774]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:53.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:52 smithi204 ceph-mon[30291]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:53.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:53 smithi017 ceph-mon[29862]: from='client.? 172.21.15.17:0/158635898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:53.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:53 smithi001 ceph-mon[25774]: from='client.? 172.21.15.17:0/158635898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:54.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:53 smithi204 ceph-mon[30291]: from='client.? 172.21.15.17:0/158635898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:17:54.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:54 smithi017 ceph-mon[29862]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:54.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:54 smithi001 ceph-mon[25774]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:55.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:54 smithi204 ceph-mon[30291]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:56.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:56 smithi017 ceph-mon[29862]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:56.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:56 smithi001 ceph-mon[25774]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:57.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:56 smithi204 ceph-mon[30291]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:58.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:58 smithi017 ceph-mon[29862]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:58.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:58 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:17:58.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:58 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:58.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:17:58 smithi017 ceph-mon[29862]: Deploying daemon osd.1 on smithi017 2024-03-11T21:17:58.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:58 smithi001 ceph-mon[25774]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:58.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:58 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:17:58.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:58 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:58.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:17:58 smithi001 ceph-mon[25774]: Deploying daemon osd.1 on smithi017 2024-03-11T21:17:59.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:58 smithi204 ceph-mon[30291]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:17:59.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:58 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:17:59.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:58 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:59.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:17:58 smithi204 ceph-mon[30291]: Deploying daemon osd.1 on smithi017 2024-03-11T21:18:00.762 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:00 smithi017 ceph-mon[29862]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:00.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:00 smithi001 ceph-mon[25774]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:01.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:00 smithi204 ceph-mon[30291]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:02.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:02 smithi017 ceph-mon[29862]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:02.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:02 smithi001 ceph-mon[25774]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:03.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:02 smithi204 ceph-mon[30291]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:04.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:04 smithi017 ceph-mon[29862]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:04.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:04 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:04.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:04 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:04.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:04 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:04 smithi001 ceph-mon[25774]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:04 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:04 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:04 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:05.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:04 smithi204 ceph-mon[30291]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:05.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:04 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:05.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:04 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:05.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:04 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:06.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:06 smithi017 ceph-mon[29862]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:06.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:06 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:06.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:06 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:06.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:06 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:06.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:06 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:06.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:06 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:06.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:06 smithi001 ceph-mon[25774]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:06.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:06 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:06.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:06 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:06.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:06 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:06.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:06 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:06.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:06 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:06 smithi204 ceph-mon[30291]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:07.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:06 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:06 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:06 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:07.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:06 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:07.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:06 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.338 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 1 on host 'smithi017' 2024-03-11T21:18:07.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:07 smithi017 ceph-mon[29862]: from='osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:18:07.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:07 smithi017 ceph-mon[29862]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:18:07.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:07 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:07 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:07 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:07.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:07 smithi001 ceph-mon[25774]: from='osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:18:07.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:07 smithi001 ceph-mon[25774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:18:07.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:07.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:07 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:08.018 DEBUG:teuthology.orchestra.run.smithi017:osd.1> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:18:08.021 INFO:tasks.cephadm:Deploying osd.2 on smithi204 with /dev/nvme4n1... 2024-03-11T21:18:08.021 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:18:08.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:07 smithi204 ceph-mon[30291]: from='osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:18:08.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:07 smithi204 ceph-mon[30291]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:18:08.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:07 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:08.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:07 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:08.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:07 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:08.269 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:08.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-mon[29862]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:08.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-mon[29862]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:18:08.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-mon[29862]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:18:08.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:08.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-mon[29862]: from='osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-11T21:18:08.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-mon[29862]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-11T21:18:08.926 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1[34424]: 2024-03-11T21:18:08.694+0000 7f76068ed700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T21:18:08.927 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:08 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1[34424]: 2024-03-11T21:18:08.697+0000 7f76026e0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:18:08.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:08 smithi001 ceph-mon[25774]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:08.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:08 smithi001 ceph-mon[25774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:18:08.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:08 smithi001 ceph-mon[25774]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:18:08.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:08 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:08.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:08 smithi001 ceph-mon[25774]: from='osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-11T21:18:08.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:08 smithi001 ceph-mon[25774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-11T21:18:09.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:08 smithi204 ceph-mon[30291]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:09.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:08 smithi204 ceph-mon[30291]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:18:09.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:08 smithi204 ceph-mon[30291]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:18:09.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:08 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:09.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:08 smithi204 ceph-mon[30291]: from='osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-11T21:18:09.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:08 smithi204 ceph-mon[30291]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-11T21:18:09.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:09 smithi001 ceph-mon[25774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-11T21:18:09.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:09 smithi001 ceph-mon[25774]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:18:09.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:09 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:09.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:09 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:09 smithi204 ceph-mon[30291]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-11T21:18:10.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:09 smithi204 ceph-mon[30291]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:18:10.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:09 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:09 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:09 smithi017 ceph-mon[29862]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-11T21:18:10.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:09 smithi017 ceph-mon[29862]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:18:10.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:09 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:09 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.241 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:18:10.241 INFO:teuthology.orchestra.run.smithi204.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:18:10.241 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:18:10.241 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-03-11T21:18:10.242 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-03-11T21:18:10.242 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122966 s, 853 MB/s 2024-03-11T21:18:10.242 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-03-11T21:18:10.850 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:10 smithi204 ceph-mon[30291]: purged_snaps scrub starts 2024-03-11T21:18:10.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:10 smithi204 ceph-mon[30291]: purged_snaps scrub ok 2024-03-11T21:18:10.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:10 smithi204 ceph-mon[30291]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:10.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:10 smithi204 ceph-mon[30291]: osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023] boot 2024-03-11T21:18:10.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:10 smithi204 ceph-mon[30291]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:18:10.851 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:10 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.872 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:10 smithi017 ceph-mon[29862]: purged_snaps scrub starts 2024-03-11T21:18:10.872 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:10 smithi017 ceph-mon[29862]: purged_snaps scrub ok 2024-03-11T21:18:10.872 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:10 smithi017 ceph-mon[29862]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:10.872 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:10 smithi017 ceph-mon[29862]: osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023] boot 2024-03-11T21:18:10.872 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:10 smithi017 ceph-mon[29862]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:18:10.872 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:10 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:10.876 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi204:/dev/nvme4n1 2024-03-11T21:18:10.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:10 smithi001 ceph-mon[25774]: purged_snaps scrub starts 2024-03-11T21:18:10.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:10 smithi001 ceph-mon[25774]: purged_snaps scrub ok 2024-03-11T21:18:10.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:10 smithi001 ceph-mon[25774]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:18:10.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:10 smithi001 ceph-mon[25774]: osd.1 [v2:172.21.15.17:6800/4026854023,v1:172.21.15.17:6801/4026854023] boot 2024-03-11T21:18:10.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:10 smithi001 ceph-mon[25774]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:18:10.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:10 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:18:11.152 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:11.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:18:11.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: Detected new or changed devices on smithi017 2024-03-11T21:18:11.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.955 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: Adjusting osd_memory_target on smithi017 to 17093M 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: Detected new or changed devices on smithi017 2024-03-11T21:18:11.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: Adjusting osd_memory_target on smithi017 to 17093M 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:11.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:12.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:18:12.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: Detected new or changed devices on smithi017 2024-03-11T21:18:12.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:12.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:12.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:12.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: Adjusting osd_memory_target on smithi017 to 17093M 2024-03-11T21:18:12.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:12.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:12.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:12.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:13.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:12 smithi204 ceph-mon[30291]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:13.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:12 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:18:13.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:12 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:18:13.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:12 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:13.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:12 smithi017 ceph-mon[29862]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:13.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:12 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:18:13.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:12 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:18:13.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:12 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:13.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:12 smithi001 ceph-mon[25774]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:13.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:12 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:18:13.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:12 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:18:13.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:12 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:14.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:13 smithi204 ceph-mon[30291]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:14.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:13 smithi017 ceph-mon[29862]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:14.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:13 smithi001 ceph-mon[25774]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:15.305 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:14 smithi204 ceph-mon[30291]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:15.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:14 smithi017 ceph-mon[29862]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:15.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:14 smithi001 ceph-mon[25774]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:16.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:15 smithi204 ceph-mon[30291]: from='client.? 172.21.15.204:0/1908597507' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]: dispatch 2024-03-11T21:18:16.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:15 smithi204 ceph-mon[30291]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]: dispatch 2024-03-11T21:18:16.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:15 smithi204 ceph-mon[30291]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]': finished 2024-03-11T21:18:16.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:15 smithi204 ceph-mon[30291]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:18:16.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:15 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:16.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:15 smithi017 ceph-mon[29862]: from='client.? 172.21.15.204:0/1908597507' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]: dispatch 2024-03-11T21:18:16.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:15 smithi017 ceph-mon[29862]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]: dispatch 2024-03-11T21:18:16.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:15 smithi017 ceph-mon[29862]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]': finished 2024-03-11T21:18:16.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:15 smithi017 ceph-mon[29862]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:18:16.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:15 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:16.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:15 smithi001 ceph-mon[25774]: from='client.? 172.21.15.204:0/1908597507' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]: dispatch 2024-03-11T21:18:16.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:15 smithi001 ceph-mon[25774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]: dispatch 2024-03-11T21:18:16.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:15 smithi001 ceph-mon[25774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3eeb574f-171a-4924-ab3b-82e04fa37c55"}]': finished 2024-03-11T21:18:16.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:15 smithi001 ceph-mon[25774]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:18:16.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:15 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:17.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:16 smithi204 ceph-mon[30291]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:17.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:16 smithi204 ceph-mon[30291]: from='client.? 172.21.15.204:0/1589248208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:18:17.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:16 smithi017 ceph-mon[29862]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:17.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:16 smithi017 ceph-mon[29862]: from='client.? 172.21.15.204:0/1589248208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:18:17.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:16 smithi001 ceph-mon[25774]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:17.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:16 smithi001 ceph-mon[25774]: from='client.? 172.21.15.204:0/1589248208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:18:19.242 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:18 smithi017 ceph-mon[29862]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:19.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:18 smithi204 ceph-mon[30291]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:19.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:18 smithi001 ceph-mon[25774]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:21.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:20 smithi204 ceph-mon[30291]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:21.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:20 smithi017 ceph-mon[29862]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:21.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:20 smithi001 ceph-mon[25774]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:23.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:23 smithi204 ceph-mon[30291]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:23.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:18:23.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:23 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:23 smithi017 ceph-mon[29862]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:18:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:23 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:23.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:23 smithi001 ceph-mon[25774]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:23.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:18:23.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:23 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:24.185 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:24 smithi204 ceph-mon[30291]: Deploying daemon osd.2 on smithi204 2024-03-11T21:18:24.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:24 smithi017 ceph-mon[29862]: Deploying daemon osd.2 on smithi204 2024-03-11T21:18:24.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:24 smithi001 ceph-mon[25774]: Deploying daemon osd.2 on smithi204 2024-03-11T21:18:25.166 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:25 smithi204 ceph-mon[30291]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:25.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:25 smithi017 ceph-mon[29862]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:25.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:25 smithi001 ceph-mon[25774]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:27.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:27 smithi204 ceph-mon[30291]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:27.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:27 smithi017 ceph-mon[29862]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:27.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:27 smithi001 ceph-mon[25774]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:29.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:29 smithi204 ceph-mon[30291]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:29.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:29 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:29.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:29 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:29.425 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:29 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:29.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:29 smithi001 ceph-mon[25774]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:29.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:29 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:29.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:29 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:29.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:29 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:29.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:29 smithi017 ceph-mon[29862]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:29.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:29 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:29.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:29 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:29.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:29 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:31.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:31 smithi001 ceph-mon[25774]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:31.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:31.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:31.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:31 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:31 smithi204 ceph-mon[30291]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:31.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:31.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:31.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:31 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:31 smithi017 ceph-mon[29862]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:31.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:31.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:31 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:31.943 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 2 on host 'smithi204' 2024-03-11T21:18:32.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:32 smithi204 ceph-mon[30291]: from='osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:32.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:32 smithi204 ceph-mon[30291]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:32.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:32 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:32.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:32 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:32.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:32 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:32.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:32 smithi001 ceph-mon[25774]: from='osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:32.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:32 smithi001 ceph-mon[25774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:32.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:32 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:32.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:32 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:32.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:32 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:32.572 DEBUG:teuthology.orchestra.run.smithi204:osd.2> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T21:18:32.574 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-11T21:18:32.574 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:18:32.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:32 smithi017 ceph-mon[29862]: from='osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:32.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:32 smithi017 ceph-mon[29862]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:18:32.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:32 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:32.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:32 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:32.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:32 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:32.826 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:33.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:33 smithi001 ceph-mon[25774]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:33.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:33 smithi001 ceph-mon[25774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:18:33.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:33 smithi001 ceph-mon[25774]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:18:33.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:33 smithi001 ceph-mon[25774]: from='osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-11T21:18:33.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:33 smithi001 ceph-mon[25774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-11T21:18:33.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:33 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:33.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-mon[30291]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:33.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-mon[30291]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:18:33.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-mon[30291]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:18:33.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-mon[30291]: from='osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-11T21:18:33.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-mon[30291]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-11T21:18:33.437 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:33.437 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2[35570]: 2024-03-11T21:18:33.207+0000 7fd652a80700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T21:18:33.437 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:33 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2[35570]: 2024-03-11T21:18:33.209+0000 7fd64e873700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:18:33.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:33 smithi017 ceph-mon[29862]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:33.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:33 smithi017 ceph-mon[29862]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:18:33.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:33 smithi017 ceph-mon[29862]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:18:33.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:33 smithi017 ceph-mon[29862]: from='osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-11T21:18:33.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:33 smithi017 ceph-mon[29862]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-03-11T21:18:33.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:33 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.257 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:34 smithi001 ceph-mon[25774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-11T21:18:34.257 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:34 smithi001 ceph-mon[25774]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:18:34.258 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:34 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.258 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:34 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:34 smithi204 ceph-mon[30291]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-11T21:18:34.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:34 smithi204 ceph-mon[30291]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:18:34.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:34 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:34 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:34 smithi017 ceph-mon[29862]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-03-11T21:18:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:34 smithi017 ceph-mon[29862]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:18:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:34 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:34 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:34.720 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:18:35.452 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: purged_snaps scrub starts 2024-03-11T21:18:35.453 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: purged_snaps scrub ok 2024-03-11T21:18:35.453 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:35.453 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654] boot 2024-03-11T21:18:35.453 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:18:35.453 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:35.453 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:35 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/754372817' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:18:35.487 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1710191914,"num_in_osds":3,"osd_in_since":1710191895,"num_remapped_pgs":0} 2024-03-11T21:18:35.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: purged_snaps scrub starts 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: purged_snaps scrub ok 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654] boot 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:35.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:35 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/754372817' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:18:35.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: purged_snaps scrub starts 2024-03-11T21:18:35.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: purged_snaps scrub ok 2024-03-11T21:18:35.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:18:35.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: osd.2 [v2:172.21.15.204:6800/3884142654,v1:172.21.15.204:6801/3884142654] boot 2024-03-11T21:18:35.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:18:35.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:18:35.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:35 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/754372817' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:18:35.733 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:36.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:18:36.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:36.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:36.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:36.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:36.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:36 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:18:36.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:36.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:36.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:36.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:36.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:36 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:18:36.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:36.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:18:36.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:36.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:36.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:36.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:36 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:37.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:37 smithi001 ceph-mon[25774]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:37.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:37 smithi001 ceph-mon[25774]: Detected new or changed devices on smithi204 2024-03-11T21:18:37.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:37 smithi001 ceph-mon[25774]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-11T21:18:37.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:37.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:37 smithi001 ceph-mon[25774]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:18:37.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:37 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:37.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:37 smithi204 ceph-mon[30291]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:37.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:37 smithi204 ceph-mon[30291]: Detected new or changed devices on smithi204 2024-03-11T21:18:37.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:37 smithi204 ceph-mon[30291]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-11T21:18:37.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:37 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:37.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:37 smithi204 ceph-mon[30291]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:18:37.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:37 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:37 smithi017 ceph-mon[29862]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:37 smithi017 ceph-mon[29862]: Detected new or changed devices on smithi204 2024-03-11T21:18:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:37 smithi017 ceph-mon[29862]: Adjusting osd_memory_target on smithi204 to 17088M 2024-03-11T21:18:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:37 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:37 smithi017 ceph-mon[29862]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:18:37.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:37 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:18:37.746 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:18:37.746 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"415704e2-dfec-11ee-95c8-87774f69a715","created":"2024-03-11T21:15:04.575022+0000","modified":"2024-03-11T21:18:37.222962+0000","last_up_change":"2024-03-11T21:18:34.204922+0000","last_in_change":"2024-03-11T21:18:15.636704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:35.719798+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6228c6e6-acf8-40c7-8960-b8591761b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6803","nonce":3615798761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6805","nonce":3615798761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6809","nonce":3615798761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6807","nonce":3615798761}]},"public_addr":"172.21.15.1:6803/3615798761","cluster_addr":"172.21.15.1:6805/3615798761","heartbeat_back_addr":"172.21.15.1:6809/3615798761","heartbeat_front_addr":"172.21.15.1:6807/3615798761","state":["exists","up"]},{"osd":1,"uuid":"1b709121-8dd2-46f8-b98f-5cbf1a181d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6801","nonce":4026854023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6803","nonce":4026854023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6807","nonce":4026854023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6805","nonce":4026854023}]},"public_addr":"172.21.15.17:6801/4026854023","cluster_addr":"172.21.15.17:6803/4026854023","heartbeat_back_addr":"172.21.15.17:6807/4026854023","heartbeat_front_addr":"172.21.15.17:6805/4026854023","state":["exists","up"]},{"osd":2,"uuid":"3eeb574f-171a-4924-ab3b-82e04fa37c55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6801","nonce":3884142654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6803","nonce":3884142654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6807","nonce":3884142654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6805","nonce":3884142654}]},"public_addr":"172.21.15.204:6801/3884142654","cluster_addr":"172.21.15.204:6803/3884142654","heartbeat_back_addr":"172.21.15.204:6807/3884142654","heartbeat_front_addr":"172.21.15.204:6805/3884142654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:17:43.680369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:07.829196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:32.281524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2433772781":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/467381966":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/1484051585":"2024-03-12T21:16:07.372673+0000","172.21.15.1:6801/479711850":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/174722667":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6800/4179260149":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6801/4179260149":"2024-03-12T21:15:40.353030+0000","172.21.15.1:0/1697036669":"2024-03-12T21:15:40.353030+0000","172.21.15.1:0/317278207":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6800/479711850":"2024-03-12T21:16:07.372673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:18:37.828 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50388]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:18:37.828 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50388]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:37.828 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50388]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:37.828 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50388]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:38.206 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50405]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:18:38.206 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:38.206 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50405]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:38.206 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:18:37 smithi001 sudo[50405]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:38.350 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:18:35.719798+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T21:18:38.351 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T21:18:38.372 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39312]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:18:38.372 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39312]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:38.372 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39312]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:38.372 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39312]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:38.373 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:38 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:38.373 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:38 smithi017 ceph-mon[29862]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:18:38.373 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:38 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/944735237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:18:38.373 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:38 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:38.373 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:38 smithi017 ceph-mon[29862]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:18:38.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:38 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:38.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:38 smithi204 ceph-mon[30291]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:18:38.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:38 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/944735237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:18:38.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:38 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:38.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:38 smithi204 ceph-mon[30291]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:18:38.591 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:38.675 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39315]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:18:38.675 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39315]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:38.675 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39315]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:38.675 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:18:38 smithi017 sudo[39315]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:38.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:38 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:18:38.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:38 smithi001 ceph-mon[25774]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:18:38.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:38 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/944735237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:18:38.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:38 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:38.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:38 smithi001 ceph-mon[25774]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:18:38.890 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40423]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:18:38.890 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40423]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:38.890 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40423]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:38.890 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40423]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:39.302 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:18:39.302 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:39.302 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40426]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:39.302 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:18:38 smithi204 sudo[40426]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:39.434 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50526]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:18:39.434 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50526]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:39.434 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50526]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:39.435 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50526]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:39.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 sudo[50529]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 ceph-mon[25774]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:39.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:39 smithi001 ceph-mon[25774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:39.714 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 ceph-mon[30291]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:39.714 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:39.714 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:39.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 ceph-mon[30291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:39.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:18:39.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:39.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:39.715 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40429]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:39.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:39 smithi017 ceph-mon[29862]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:39.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:39 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:39.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:39 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:39.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:39 smithi017 ceph-mon[29862]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:40.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40432]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:18:40.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:40.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40432]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:40.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:39 smithi204 sudo[40432]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:40.499 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39318]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:18:40.499 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39318]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:40.500 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39318]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:40.500 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39318]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:40.532 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-03-11T21:18:40.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:40.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:40.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.804 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:40 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.894 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:40.894 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.896 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:40 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39321]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:18:40.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39321]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:18:40.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39321]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:18:40.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 sudo[39321]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:40.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:40.927 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:18:40.927 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:18:40.927 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:40 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:18:41.191 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi001 2024-03-11T21:18:41.191 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi001=foo.a' 2024-03-11T21:18:41.440 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:41.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:41 smithi204 ceph-mon[30291]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:41.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:41 smithi204 ceph-mon[30291]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:41.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:41 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:41.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:41 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/3214322405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:18:41.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:41 smithi017 ceph-mon[29862]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:41.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:41 smithi017 ceph-mon[29862]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:41.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:41 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:41.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:41 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/3214322405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:18:41.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:41 smithi001 ceph-mon[25774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:18:41.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:41 smithi001 ceph-mon[25774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:18:41.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:41 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:41.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:41 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/3214322405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:18:42.550 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:42 smithi204 ceph-mon[30291]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:42.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:42 smithi017 ceph-mon[29862]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:42.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:42 smithi001 ceph-mon[25774]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:43.212 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled rgw.foo.a update... 2024-03-11T21:18:43.877 DEBUG:teuthology.orchestra.run.smithi001:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T21:18:43.880 INFO:tasks.cephadm:Adding prometheus.a on smithi017 2024-03-11T21:18:43.881 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi017=a' 2024-03-11T21:18:44.120 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi001=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: Saving service rgw.foo.a spec with placement smithi001=foo.a;count:1 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:18:44.476 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.477 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:44 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:44.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi001=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:44.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: Saving service rgw.foo.a spec with placement smithi001=foo.a;count:1 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:44 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:44.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi001=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:44.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: Saving service rgw.foo.a spec with placement smithi001=foo.a;count:1 2024-03-11T21:18:44.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:44.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:44 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:45.490 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:45 smithi204 ceph-mon[30291]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:45.490 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:45 smithi204 ceph-mon[30291]: Deploying daemon rgw.foo.a on smithi001 2024-03-11T21:18:45.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:45 smithi001 ceph-mon[25774]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:45.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:45 smithi001 ceph-mon[25774]: Deploying daemon rgw.foo.a on smithi001 2024-03-11T21:18:45.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:45 smithi017 ceph-mon[29862]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:18:45.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:45 smithi017 ceph-mon[29862]: Deploying daemon rgw.foo.a on smithi001 2024-03-11T21:18:45.816 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled prometheus update... 2024-03-11T21:18:46.202 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 systemd[1]: Starting Ceph rgw.foo.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:18:46.402 DEBUG:teuthology.orchestra.run.smithi017:prometheus.a> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T21:18:46.405 INFO:tasks.cephadm:Adding node-exporter.a on smithi001 2024-03-11T21:18:46.405 INFO:tasks.cephadm:Adding node-exporter.b on smithi017 2024-03-11T21:18:46.405 INFO:tasks.cephadm:Adding node-exporter.c on smithi204 2024-03-11T21:18:46.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi001=a;smithi017=b;smithi204=c' 2024-03-11T21:18:46.457 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 podman[50935]: 2024-03-11 21:18:46.217616786 +0000 UTC m=+0.111744855 container create 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-11T21:18:46.457 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 podman[50935]: 2024-03-11 21:18:46.123592873 +0000 UTC m=+0.017720948 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:18:46.457 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 podman[50935]: 2024-03-11 21:18:46.42856613 +0000 UTC m=+0.322694209 container init 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307) 2024-03-11T21:18:46.457 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 podman[50935]: 2024-03-11 21:18:46.434332575 +0000 UTC m=+0.328460642 container start 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T21:18:46.637 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:46.815 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 bash[50935]: 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c 2024-03-11T21:18:46.815 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:18:46 smithi001 systemd[1]: Started Ceph rgw.foo.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: Saving service prometheus spec with placement smithi017=a;count:1 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:46 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-11T21:18:47.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:47.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: Saving service prometheus spec with placement smithi017=a;count:1 2024-03-11T21:18:47.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:46 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-11T21:18:47.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:47.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: Saving service prometheus spec with placement smithi017=a;count:1 2024-03-11T21:18:47.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:47.207 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:46 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:48.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:47 smithi204 ceph-mon[30291]: Saving service rgw.foo.a spec with placement smithi001=foo.a;count:1 2024-03-11T21:18:48.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:47 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:48.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:47 smithi204 ceph-mon[30291]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:18:48.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:47 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2280039277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:18:48.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:47 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:48.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:47 smithi017 ceph-mon[29862]: Saving service rgw.foo.a spec with placement smithi001=foo.a;count:1 2024-03-11T21:18:48.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:47 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:48.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:47 smithi017 ceph-mon[29862]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:18:48.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:47 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2280039277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:18:48.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:47 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:48.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:47 smithi001 ceph-mon[25774]: Saving service rgw.foo.a spec with placement smithi001=foo.a;count:1 2024-03-11T21:18:48.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:48.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:47 smithi001 ceph-mon[25774]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:18:48.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:47 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2280039277' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:18:48.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:47 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:48.546 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled node-exporter update... 2024-03-11T21:18:49.158 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: pgmap v93: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2280039277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:49.159 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:48 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:49.184 DEBUG:teuthology.orchestra.run.smithi001:node-exporter.a> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T21:18:49.187 DEBUG:teuthology.orchestra.run.smithi017:node-exporter.b> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T21:18:49.190 DEBUG:teuthology.orchestra.run.smithi204:node-exporter.c> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-11T21:18:49.192 INFO:tasks.cephadm:Adding alertmanager.a on smithi001 2024-03-11T21:18:49.193 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi001=a' 2024-03-11T21:18:49.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: pgmap v93: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:18:49.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:18:49.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2280039277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:18:49.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:18:49.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:49.426 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:48 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: pgmap v93: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2280039277' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:49.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:49.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:48 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:49.465 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:50.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:50 smithi204 ceph-mon[30291]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi001=a;smithi017=b;smithi204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:50.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:50 smithi204 ceph-mon[30291]: Saving service node-exporter spec with placement smithi001=a;smithi017=b;smithi204=c;count:3 2024-03-11T21:18:50.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:50 smithi204 ceph-mon[30291]: Checking dashboard <-> RGW credentials 2024-03-11T21:18:50.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:50 smithi204 ceph-mon[30291]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:18:50.530 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:50 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:18:50.531 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:50 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:18:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:50 smithi017 ceph-mon[29862]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi001=a;smithi017=b;smithi204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:50 smithi017 ceph-mon[29862]: Saving service node-exporter spec with placement smithi001=a;smithi017=b;smithi204=c;count:3 2024-03-11T21:18:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:50 smithi017 ceph-mon[29862]: Checking dashboard <-> RGW credentials 2024-03-11T21:18:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:50 smithi017 ceph-mon[29862]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:18:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:50 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:18:50.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:50 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:18:50.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:50 smithi001 ceph-mon[25774]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi001=a;smithi017=b;smithi204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:50.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:50 smithi001 ceph-mon[25774]: Saving service node-exporter spec with placement smithi001=a;smithi017=b;smithi204=c;count:3 2024-03-11T21:18:50.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:50 smithi001 ceph-mon[25774]: Checking dashboard <-> RGW credentials 2024-03-11T21:18:50.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:50 smithi001 ceph-mon[25774]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:18:50.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:50 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:18:50.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:50 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:18:51.297 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled alertmanager update... 2024-03-11T21:18:51.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:51 smithi204 ceph-mon[30291]: pgmap v96: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:18:51.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:51 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:18:51.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:51 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:18:51.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:51 smithi204 ceph-mon[30291]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:18:51.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:51 smithi017 ceph-mon[29862]: pgmap v96: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:18:51.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:51 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:18:51.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:51 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:18:51.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:51 smithi017 ceph-mon[29862]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:18:51.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:51 smithi001 ceph-mon[25774]: pgmap v96: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T21:18:51.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:51 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:18:51.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:51 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:18:51.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:51 smithi001 ceph-mon[25774]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:18:51.870 DEBUG:teuthology.orchestra.run.smithi001:alertmanager.a> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T21:18:51.874 INFO:tasks.cephadm:Adding grafana.a on smithi017 2024-03-11T21:18:51.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi017=a' 2024-03-11T21:18:52.106 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:18:52.522 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:18:52.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:18:52.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:18:52.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi001=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:52.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: Saving service alertmanager spec with placement smithi001=a;count:1 2024-03-11T21:18:52.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:52.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:52 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:18:52.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:18:52.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:18:52.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:18:52.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi001=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:52.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: Saving service alertmanager spec with placement smithi001=a;count:1 2024-03-11T21:18:52.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:52.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:52 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi001=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: Saving service alertmanager spec with placement smithi001=a;count:1 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:52.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:52 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:18:53.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:53 smithi204 ceph-mon[30291]: pgmap v99: 97 pgs: 31 creating+peering, 21 unknown, 45 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:18:53.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:53 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:18:53.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:53 smithi204 ceph-mon[30291]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:18:53.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:53 smithi017 ceph-mon[29862]: pgmap v99: 97 pgs: 31 creating+peering, 21 unknown, 45 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:18:53.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:53 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:18:53.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:53 smithi017 ceph-mon[29862]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:18:53.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:53 smithi001 ceph-mon[25774]: pgmap v99: 97 pgs: 31 creating+peering, 21 unknown, 45 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:18:53.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:53 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:18:53.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:53 smithi001 ceph-mon[25774]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:18:53.889 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled grafana update... 2024-03-11T21:18:54.491 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:54 smithi204 ceph-mon[30291]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:18:54.491 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:54 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:18:54.491 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:54 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:18:54.491 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:54 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:54.491 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:54 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:18:54.517 DEBUG:teuthology.orchestra.run.smithi017:grafana.a> sudo journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T21:18:54.520 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T21:18:54.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:18:54.545 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:54 smithi017 ceph-mon[29862]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:18:54.545 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:54 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:18:54.546 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:54 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:18:54.546 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:54 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:54.546 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:54 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:18:54.548 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:54 smithi001 ceph-mon[25774]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:18:54.548 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:54 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:18:54.548 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:54 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:18:54.548 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:54 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:54.548 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:54 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:18:54.768 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:18:55.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: pgmap v102: 129 pgs: 31 creating+peering, 53 unknown, 45 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: Saving service grafana spec with placement smithi017=a;count:1 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:18:55.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:55 smithi001 ceph-mon[25774]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: pgmap v102: 129 pgs: 31 creating+peering, 53 unknown, 45 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: Saving service grafana spec with placement smithi017=a;count:1 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:18:55.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:18:55.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:18:55.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:18:55.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:55 smithi204 ceph-mon[30291]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:18:55.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: pgmap v102: 129 pgs: 31 creating+peering, 53 unknown, 45 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:18:55.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:18:55.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: Saving service grafana spec with placement smithi017=a;count:1 2024-03-11T21:18:55.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:18:55.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:18:55.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:18:55.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:18:55.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/245934873' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:18:55.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1077723461' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:18:55.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:55 smithi017 ceph-mon[29862]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:18:56.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: pgmap v105: 129 pgs: 11 creating+peering, 10 unknown, 108 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:18:56.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: Deploying daemon prometheus.a on smithi017 2024-03-11T21:18:56.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:18:56.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:56 smithi017 ceph-mon[29862]: Cluster is now healthy 2024-03-11T21:18:56.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: pgmap v105: 129 pgs: 11 creating+peering, 10 unknown, 108 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:18:56.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: Deploying daemon prometheus.a on smithi017 2024-03-11T21:18:56.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:18:56.707 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:56 smithi001 ceph-mon[25774]: Cluster is now healthy 2024-03-11T21:18:56.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: pgmap v105: 129 pgs: 11 creating+peering, 10 unknown, 108 active+clean; 578 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:18:56.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:18:56.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: Deploying daemon prometheus.a on smithi017 2024-03-11T21:18:56.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:18:56.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:56 smithi204 ceph-mon[30291]: Cluster is now healthy 2024-03-11T21:18:56.884 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-03-11T21:18:56.885 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBAde9lluqPNRAADBYe8wqCz4OvC3Fr5ab/yw== 2024-03-11T21:18:57.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:57 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/221009884' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:18:57.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:57 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/221009884' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:18:57.527 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:18:57.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T21:18:57.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T21:18:57.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:18:57.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:57 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/221009884' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:18:57.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:57 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/221009884' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:18:57.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:57 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/221009884' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:18:57.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:57 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/221009884' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:18:57.864 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:18:59.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:18:58 smithi204 ceph-mon[30291]: pgmap v106: 129 pgs: 129 active+clean; 579 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 33 KiB/s rd, 2.5 KiB/s wr, 80 op/s 2024-03-11T21:18:59.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:18:58 smithi017 ceph-mon[29862]: pgmap v106: 129 pgs: 129 active+clean; 579 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 33 KiB/s rd, 2.5 KiB/s wr, 80 op/s 2024-03-11T21:18:59.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:18:58 smithi001 ceph-mon[25774]: pgmap v106: 129 pgs: 129 active+clean; 579 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 33 KiB/s rd, 2.5 KiB/s wr, 80 op/s 2024-03-11T21:19:00.363 INFO:teuthology.orchestra.run.smithi017.stdout:[client.1] 2024-03-11T21:19:00.364 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQBEde9lVqlcFhAANiLsfvk1N8o8m+9mDLg7WQ== 2024-03-11T21:19:01.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:00 smithi017 ceph-mon[29862]: pgmap v107: 129 pgs: 129 active+clean; 579 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-03-11T21:19:01.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:00 smithi017 ceph-mon[29862]: from='client.? 172.21.15.17:0/612738829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:01.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:00 smithi017 ceph-mon[29862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:01.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:00 smithi017 ceph-mon[29862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:19:01.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:00 smithi001 ceph-mon[25774]: pgmap v107: 129 pgs: 129 active+clean; 579 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-03-11T21:19:01.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:00 smithi001 ceph-mon[25774]: from='client.? 172.21.15.17:0/612738829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:01.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:00 smithi001 ceph-mon[25774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:01.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:00 smithi001 ceph-mon[25774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:19:01.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:00 smithi204 ceph-mon[30291]: pgmap v107: 129 pgs: 129 active+clean; 579 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 28 KiB/s rd, 2.1 KiB/s wr, 67 op/s 2024-03-11T21:19:01.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:00 smithi204 ceph-mon[30291]: from='client.? 172.21.15.17:0/612738829' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:01.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:00 smithi204 ceph-mon[30291]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:01.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:00 smithi204 ceph-mon[30291]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:19:01.778 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:19:01.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T21:19:01.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T21:19:01.820 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:19:02.068 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:19:03.172 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:02 smithi204 ceph-mon[30291]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T21:19:03.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:02 smithi017 ceph-mon[29862]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T21:19:03.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:02 smithi001 ceph-mon[25774]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T21:19:03.987 INFO:teuthology.orchestra.run.smithi204.stdout:[client.2] 2024-03-11T21:19:03.987 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBHde9lyTKFOxAA3Aj7RZT6NfoIXWhvB5o4SA== 2024-03-11T21:19:04.587 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:19:04.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-11T21:19:04.588 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-11T21:19:04.629 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T21:19:04.629 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T21:19:04.629 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T21:19:04.881 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:04 smithi001 ceph-mon[25774]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-03-11T21:19:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:04 smithi001 ceph-mon[25774]: from='client.? 172.21.15.204:0/1268320055' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:04 smithi001 ceph-mon[25774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:04.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:04 smithi001 ceph-mon[25774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:19:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:04 smithi017 ceph-mon[29862]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-03-11T21:19:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:04 smithi017 ceph-mon[29862]: from='client.? 172.21.15.204:0/1268320055' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:04 smithi017 ceph-mon[29862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:04 smithi017 ceph-mon[29862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:19:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:04 smithi204 ceph-mon[30291]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-03-11T21:19:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:04 smithi204 ceph-mon[30291]: from='client.? 172.21.15.204:0/1268320055' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:04 smithi204 ceph-mon[30291]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:19:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:04 smithi204 ceph-mon[30291]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:19:06.876 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:19:07.156 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:06 smithi017 ceph-mon[29862]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 110 op/s 2024-03-11T21:19:07.156 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:06 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2434525168' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:19:07.191 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:06 smithi001 ceph-mon[25774]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 110 op/s 2024-03-11T21:19:07.192 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:06 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2434525168' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:19:07.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:06 smithi204 ceph-mon[30291]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 110 op/s 2024-03-11T21:19:07.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:06 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2434525168' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:19:07.538 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":1938662680},{"type":"v1","addr":"172.21.15.1:6801","nonce":1938662680}]},"active_addr":"172.21.15.1:6801/1938662680","active_change":"2024-03-11T21:16:07.372776+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.1:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3811557858}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2835875501}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":740255419}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3538057216}]}]} 2024-03-11T21:19:07.545 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T21:19:07.545 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:19:07.545 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:19:07.781 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:09.173 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:08 smithi001 ceph-mon[25774]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 93 op/s 2024-03-11T21:19:09.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:08 smithi017 ceph-mon[29862]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 93 op/s 2024-03-11T21:19:09.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:08 smithi204 ceph-mon[30291]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 93 op/s 2024-03-11T21:19:09.646 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:19:09.646 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":31,"fsid":"415704e2-dfec-11ee-95c8-87774f69a715","created":"2024-03-11T21:15:04.575022+0000","modified":"2024-03-11T21:18:55.257494+0000","last_up_change":"2024-03-11T21:18:34.204922+0000","last_in_change":"2024-03-11T21:18:15.636704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:35.719798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:18:46.720741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:18:48.554492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:18:50.505343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:18:52.564048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6228c6e6-acf8-40c7-8960-b8591761b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6803","nonce":3615798761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6805","nonce":3615798761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6809","nonce":3615798761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6807","nonce":3615798761}]},"public_addr":"172.21.15.1:6803/3615798761","cluster_addr":"172.21.15.1:6805/3615798761","heartbeat_back_addr":"172.21.15.1:6809/3615798761","heartbeat_front_addr":"172.21.15.1:6807/3615798761","state":["exists","up"]},{"osd":1,"uuid":"1b709121-8dd2-46f8-b98f-5cbf1a181d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6801","nonce":4026854023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6803","nonce":4026854023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6807","nonce":4026854023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6805","nonce":4026854023}]},"public_addr":"172.21.15.17:6801/4026854023","cluster_addr":"172.21.15.17:6803/4026854023","heartbeat_back_addr":"172.21.15.17:6807/4026854023","heartbeat_front_addr":"172.21.15.17:6805/4026854023","state":["exists","up"]},{"osd":2,"uuid":"3eeb574f-171a-4924-ab3b-82e04fa37c55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6801","nonce":3884142654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6803","nonce":3884142654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6807","nonce":3884142654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6805","nonce":3884142654}]},"public_addr":"172.21.15.204:6801/3884142654","cluster_addr":"172.21.15.204:6803/3884142654","heartbeat_back_addr":"172.21.15.204:6807/3884142654","heartbeat_front_addr":"172.21.15.204:6805/3884142654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:17:43.680369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:07.829196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:32.281524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2433772781":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/467381966":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/1484051585":"2024-03-12T21:16:07.372673+0000","172.21.15.1:6801/479711850":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/174722667":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6800/4179260149":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6801/4179260149":"2024-03-12T21:15:40.353030+0000","172.21.15.1:0/1697036669":"2024-03-12T21:15:40.353030+0000","172.21.15.1:0/317278207":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6800/479711850":"2024-03-12T21:16:07.372673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:19:09.905 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:09 smithi017 systemd[1]: Starting Ceph prometheus.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:19:10.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:09 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1212726291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:10.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:09 smithi017 podman[40220]: 2024-03-11 21:19:09.896185329 +0000 UTC m=+0.018505117 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T21:19:10.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 podman[40220]: 2024-03-11 21:19:10.001907044 +0000 UTC m=+0.124226891 container create 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:19:10.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:09 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1212726291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:10.288 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:19:10.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:19:10.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:09 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1212726291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:10.526 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:10.675 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 podman[40220]: 2024-03-11 21:19:10.237792626 +0000 UTC m=+0.360112416 container init 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 podman[40220]: 2024-03-11 21:19:10.243141402 +0000 UTC m=+0.365461188 container start 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.267Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.267Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.267Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi017 (none))" 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.267Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.267Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.268Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.268Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.269Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.269Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.272Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.272Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.421µs 2024-03-11T21:19:10.676 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.272Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.273Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.273Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.907µs wal_replay_duration=262.296µs wbl_replay_duration=291ns total_replay_duration=310.406µs 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.273Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.273Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.273Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.293Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.109591ms db_storage=792ns remote_storage=1.155µs web_handler=332ns query_engine=520ns scrape=400.227µs scrape_sd=82.637µs notify=684ns notify_sd=1.428µs rules=19.356186ms tracing=7.305µs 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.293Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:19:10.293Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 bash[40220]: 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e 2024-03-11T21:19:10.677 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:19:10 smithi017 systemd[1]: Started Ceph prometheus.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:19:11.663 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-mon[25774]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 1.9 KiB/s wr, 59 op/s 2024-03-11T21:19:11.663 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.663 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.663 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.663 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:19:11.664 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ignoring --setuser ceph since I am not root 2024-03-11T21:19:11.664 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:11 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ignoring --setgroup ceph since I am not root 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-mon[29862]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 1.9 KiB/s wr, 59 op/s 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: ignoring --setuser ceph since I am not root 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: ignoring --setgroup ceph since I am not root 2024-03-11T21:19:11.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-mgr[31337]: -- 172.21.15.17:0/2364999331 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555851bfc5a0 con 0x555851bd0800 2024-03-11T21:19:11.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-mon[30291]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 1.9 KiB/s wr, 59 op/s 2024-03-11T21:19:11.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' 2024-03-11T21:19:11.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:19:11.803 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: ignoring --setuser ceph since I am not root 2024-03-11T21:19:11.803 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: ignoring --setgroup ceph since I am not root 2024-03-11T21:19:11.803 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-mgr[32394]: -- 172.21.15.204:0/2347763843 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e214ae85a0 con 0x55e214abcc00 2024-03-11T21:19:12.255 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:12 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:12.001+0000 7ff4baf1e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:19:12.255 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:12 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:12.100+0000 7ff4baf1e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:19:12.302 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:11 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:11.975+0000 7efc6a337200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:19:12.302 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:12 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:12.067+0000 7efc6a337200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:19:12.397 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:11 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:11.994+0000 7f97ca2fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:19:12.397 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:12 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:12.089+0000 7f97ca2fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:19:12.499 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:19:12.499 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":31,"fsid":"415704e2-dfec-11ee-95c8-87774f69a715","created":"2024-03-11T21:15:04.575022+0000","modified":"2024-03-11T21:18:55.257494+0000","last_up_change":"2024-03-11T21:18:34.204922+0000","last_in_change":"2024-03-11T21:18:15.636704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:18:35.719798+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:18:46.720741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:18:48.554492+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:18:50.505343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:18:52.564048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6228c6e6-acf8-40c7-8960-b8591761b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6803","nonce":3615798761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6805","nonce":3615798761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6809","nonce":3615798761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3615798761},{"type":"v1","addr":"172.21.15.1:6807","nonce":3615798761}]},"public_addr":"172.21.15.1:6803/3615798761","cluster_addr":"172.21.15.1:6805/3615798761","heartbeat_back_addr":"172.21.15.1:6809/3615798761","heartbeat_front_addr":"172.21.15.1:6807/3615798761","state":["exists","up"]},{"osd":1,"uuid":"1b709121-8dd2-46f8-b98f-5cbf1a181d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6801","nonce":4026854023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6803","nonce":4026854023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6807","nonce":4026854023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4026854023},{"type":"v1","addr":"172.21.15.17:6805","nonce":4026854023}]},"public_addr":"172.21.15.17:6801/4026854023","cluster_addr":"172.21.15.17:6803/4026854023","heartbeat_back_addr":"172.21.15.17:6807/4026854023","heartbeat_front_addr":"172.21.15.17:6805/4026854023","state":["exists","up"]},{"osd":2,"uuid":"3eeb574f-171a-4924-ab3b-82e04fa37c55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6801","nonce":3884142654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6803","nonce":3884142654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6807","nonce":3884142654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":3884142654},{"type":"v1","addr":"172.21.15.204:6805","nonce":3884142654}]},"public_addr":"172.21.15.204:6801/3884142654","cluster_addr":"172.21.15.204:6803/3884142654","heartbeat_back_addr":"172.21.15.204:6807/3884142654","heartbeat_front_addr":"172.21.15.204:6805/3884142654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:17:43.680369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:07.829196+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:18:32.281524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2433772781":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/467381966":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/1484051585":"2024-03-12T21:16:07.372673+0000","172.21.15.1:6801/479711850":"2024-03-12T21:16:07.372673+0000","172.21.15.1:0/174722667":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6800/4179260149":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6801/4179260149":"2024-03-12T21:15:40.353030+0000","172.21.15.1:0/1697036669":"2024-03-12T21:15:40.353030+0000","172.21.15.1:0/317278207":"2024-03-12T21:15:40.353030+0000","172.21.15.1:6800/479711850":"2024-03-12T21:16:07.372673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:19:12.511 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:12 smithi001 ceph-mon[25774]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:19:12.511 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:12 smithi001 ceph-mon[25774]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:19:12.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:12 smithi017 ceph-mon[29862]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:19:12.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:12 smithi017 ceph-mon[29862]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:19:12.691 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:12 smithi204 ceph-mon[30291]: from='mgr.14152 172.21.15.1:0/1728176350' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:19:12.691 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:12 smithi204 ceph-mon[30291]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:19:13.020 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:12 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:12.706+0000 7efc6a337200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:19:13.020 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:12 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:12.798+0000 7efc6a337200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:19:13.020 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:12 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:12.940+0000 7efc6a337200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:19:13.020 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.035+0000 7efc6a337200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:19:13.073 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:12 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:12.750+0000 7f97ca2fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:19:13.073 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:12 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:12.846+0000 7f97ca2fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:19:13.073 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:12 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:12.990+0000 7f97ca2fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:19:13.074 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.087+0000 7f97ca2fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:19:13.120 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:12 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:12.796+0000 7ff4baf1e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:19:13.120 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:12 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:12.892+0000 7ff4baf1e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:19:13.121 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.036+0000 7ff4baf1e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:19:13.121 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.135+0000 7ff4baf1e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:19:13.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:19:13.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:19:13.169 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:19:13.302 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.117+0000 7efc6a337200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:19:13.398 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.227+0000 7ff4baf1e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:19:13.399 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.173+0000 7f97ca2fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:19:13.448 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:13.456 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:13.459 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:13.653 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/665188670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:13.654 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.379+0000 7efc6a337200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:19:13.654 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.489+0000 7efc6a337200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:19:13.654 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.574+0000 7efc6a337200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:19:13.654 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.668+0000 7efc6a337200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:19:13.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/665188670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:13.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.442+0000 7f97ca2fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:19:13.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.554+0000 7f97ca2fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:19:13.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.642+0000 7f97ca2fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:19:13.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/665188670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:19:13.706 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.542+0000 7ff4baf1e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:19:13.706 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.655+0000 7ff4baf1e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:19:13.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.750+0000 7ff4baf1e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:19:13.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:13.849+0000 7ff4baf1e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:19:14.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:13 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:13.845+0000 7efc6a337200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:19:14.175 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.739+0000 7f97ca2fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:19:14.175 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:13 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:13.917+0000 7f97ca2fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:19:14.282 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:14 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:14.033+0000 7ff4baf1e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:19:15.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:14 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:14.807+0000 7efc6a337200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:19:15.052 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:14 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:14.937+0000 7efc6a337200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:19:15.162 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:15 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:15.044+0000 7ff4baf1e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:19:15.174 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:14 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:14.900+0000 7f97ca2fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:19:15.175 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:15 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:15.034+0000 7f97ca2fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:19:15.454 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:15 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:15.181+0000 7ff4baf1e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:19:15.543 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:15 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:15.271+0000 7efc6a337200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:19:15.545 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:15 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:15.362+0000 7efc6a337200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:19:15.545 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:15 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:15.557+0000 7efc6a337200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:19:15.650 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:15 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:15.371+0000 7f97ca2fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:19:15.650 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:15 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:15.464+0000 7f97ca2fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:19:15.650 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:15 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:15.664+0000 7f97ca2fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:19:15.705 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:15 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:15.523+0000 7ff4baf1e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:19:15.705 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:15 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:15.618+0000 7ff4baf1e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:19:15.802 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:15 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:15.743+0000 7efc6a337200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:19:15.925 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:15 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:15.853+0000 7f97ca2fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:19:15.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:15 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:15.820+0000 7ff4baf1e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:19:16.273 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:16 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:16.026+0000 7efc6a337200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:19:16.273 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:16 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:16.288+0000 7efc6a337200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:19:16.282 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:16 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:16.013+0000 7ff4baf1e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:19:16.390 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:16 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:16.143+0000 7f97ca2fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:19:16.391 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:16 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:16.405+0000 7f97ca2fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:19:16.552 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:16 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:16.522+0000 7efc6a337200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:19:16.567 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:16 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:16.312+0000 7ff4baf1e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:19:16.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:16 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:16.648+0000 7f97ca2fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:19:16.819 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:16 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:16.582+0000 7ff4baf1e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:19:16.819 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:16 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:16.833+0000 7ff4baf1e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:19:17.552 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:17 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:17.311+0000 7efc6a337200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:19:17.706 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:17 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:17.712+0000 7ff4baf1e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:19:17.925 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:17 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:17.474+0000 7f97ca2fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:19:17.988 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:17 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:17.899+0000 7efc6a337200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:19:17.988 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:18.002+0000 7efc6a337200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:19:18.260 INFO:teuthology.orchestra.run.smithi001.stdout:55834574862 2024-03-11T21:19:18.260 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:19:18.289 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:18.229+0000 7efc6a337200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:19:18.289 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: [11/Mar/2024:21:19:18] ENGINE Bus STARTING 2024-03-11T21:19:18.363 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-mon[29862]: Standby manager daemon z restarted 2024-03-11T21:19:18.363 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-mon[29862]: Standby manager daemon z started 2024-03-11T21:19:18.363 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:19:18.363 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:19:18.364 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:19:18.364 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:19:18.364 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:18.056+0000 7f97ca2fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:19:18.364 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:18.157+0000 7f97ca2fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:19:18.364 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:18.379+0000 7f97ca2fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:18.309+0000 7ff4baf1e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:18.415+0000 7ff4baf1e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-mon[25774]: Standby manager daemon z restarted 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-mon[25774]: Standby manager daemon z started 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:19:18.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:19:18.494 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:19:18.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-mon[30291]: Standby manager daemon z restarted 2024-03-11T21:19:18.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-mon[30291]: Standby manager daemon z started 2024-03-11T21:19:18.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:19:18.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:19:18.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:19:18.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.204:0/3160690461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:19:18.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: CherryPy Checker: 2024-03-11T21:19:18.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: The Application mounted at '' has an empty config. 2024-03-11T21:19:18.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: 2024-03-11T21:19:18.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: [11/Mar/2024:21:19:18] ENGINE Serving on http://:::9283 2024-03-11T21:19:18.553 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:19:18 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z[32370]: [11/Mar/2024:21:19:18] ENGINE Bus STARTED 2024-03-11T21:19:18.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: [11/Mar/2024:21:19:18] ENGINE Bus STARTING 2024-03-11T21:19:18.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: CherryPy Checker: 2024-03-11T21:19:18.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: The Application mounted at '' has an empty config. 2024-03-11T21:19:18.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: 2024-03-11T21:19:18.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: [11/Mar/2024:21:19:18] ENGINE Serving on http://:::9283 2024-03-11T21:19:18.675 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:19:18 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x[31314]: [11/Mar/2024:21:19:18] ENGINE Bus STARTED 2024-03-11T21:19:18.782 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:18.653+0000 7ff4baf1e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:19:18.782 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:18] ENGINE Bus STARTING 2024-03-11T21:19:19.206 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: CherryPy Checker: 2024-03-11T21:19:19.206 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: The Application mounted at '' has an empty config. 2024-03-11T21:19:19.206 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:19:19.206 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:18] ENGINE Serving on http://:::9283 2024-03-11T21:19:19.206 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:18 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:18] ENGINE Bus STARTED 2024-03-11T21:19:19.323 INFO:teuthology.orchestra.run.smithi001.stdout:77309411337 2024-03-11T21:19:19.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:19:19.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: Standby manager daemon x restarted 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: Standby manager daemon x started 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: Active manager daemon y restarted 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: Activating manager daemon y 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: mgrmap e19: y(active, starting, since 0.0111827s), standbys: x, z 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:19:19.553 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: Manager daemon y is now available 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:19.554 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:19.555 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:19.555 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:19:19.555 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:19 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: Standby manager daemon x restarted 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: Standby manager daemon x started 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:19:19.595 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: Active manager daemon y restarted 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: Activating manager daemon y 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: mgrmap e19: y(active, starting, since 0.0111827s), standbys: x, z 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:19:19.596 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: Manager daemon y is now available 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:19:19.597 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:19 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: Standby manager daemon x restarted 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: Standby manager daemon x started 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.? 172.21.15.17:0/1960262866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: Active manager daemon y restarted 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: Activating manager daemon y 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: osdmap e32: 3 total, 3 up, 3 in 2024-03-11T21:19:19.700 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: mgrmap e19: y(active, starting, since 0.0111827s), standbys: x, z 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: Manager daemon y is now available 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:19.701 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:19.702 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:19.702 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:19:19.702 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:19 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:19:20.017 INFO:teuthology.orchestra.run.smithi001.stdout:34359738388 2024-03-11T21:19:20.018 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:19:20.113 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:20.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: [11/Mar/2024:21:19:19] ENGINE Bus STARTING 2024-03-11T21:19:20.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: [11/Mar/2024:21:19:19] ENGINE Serving on http://172.21.15.1:8765 2024-03-11T21:19:20.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: mgrmap e20: y(active, since 1.01428s), standbys: x, z 2024-03-11T21:19:20.925 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: [11/Mar/2024:21:19:19] ENGINE Serving on https://172.21.15.1:7150 2024-03-11T21:19:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: [11/Mar/2024:21:19:19] ENGINE Bus STARTED 2024-03-11T21:19:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:20.926 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:20 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: [11/Mar/2024:21:19:19] ENGINE Bus STARTING 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: [11/Mar/2024:21:19:19] ENGINE Serving on http://172.21.15.1:8765 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: mgrmap e20: y(active, since 1.01428s), standbys: x, z 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: [11/Mar/2024:21:19:19] ENGINE Serving on https://172.21.15.1:7150 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: [11/Mar/2024:21:19:19] ENGINE Bus STARTED 2024-03-11T21:19:20.956 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:20.957 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:20 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:21.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: [11/Mar/2024:21:19:19] ENGINE Bus STARTING 2024-03-11T21:19:21.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: [11/Mar/2024:21:19:19] ENGINE Serving on http://172.21.15.1:8765 2024-03-11T21:19:21.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: mgrmap e20: y(active, since 1.01428s), standbys: x, z 2024-03-11T21:19:21.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:21.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: [11/Mar/2024:21:19:19] ENGINE Serving on https://172.21.15.1:7150 2024-03-11T21:19:21.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: [11/Mar/2024:21:19:19] ENGINE Bus STARTED 2024-03-11T21:19:21.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:21.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:20 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:21.855 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:21.896 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:22.237 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:22 smithi001 ceph-mon[25774]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:22.237 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:22 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.237 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:22 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.237 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:22 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.237 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:22 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.237 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:22 smithi001 ceph-mon[25774]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:19:22.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:22 smithi017 ceph-mon[29862]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:22.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:22 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:22 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:22 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:22 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:22 smithi017 ceph-mon[29862]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:19:22.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:22 smithi204 ceph-mon[30291]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:22.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:22 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:22 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:22 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:22 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:22.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:22 smithi204 ceph-mon[30291]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:19:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:23 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:23 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:23 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:23 smithi017 ceph-mon[29862]: Adjusting osd_memory_target on smithi017 to 16069M 2024-03-11T21:19:23.425 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:23 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.426 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:23 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.426 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:23 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.426 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:23 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:23.426 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:23 smithi001 ceph-mon[25774]: Adjusting osd_memory_target on smithi017 to 16069M 2024-03-11T21:19:23.427 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:23 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:23 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:23 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:23.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:23 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:23.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:23 smithi204 ceph-mon[30291]: Adjusting osd_memory_target on smithi017 to 16069M 2024-03-11T21:19:23.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:23 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:24.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:24 smithi001 ceph-mon[25774]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:24.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:24 smithi001 ceph-mon[25774]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:19:24.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:24 smithi204 ceph-mon[30291]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:24.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:24 smithi204 ceph-mon[30291]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:19:24.608 INFO:teuthology.orchestra.run.smithi001.stdout:77309411338 2024-03-11T21:19:24.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:24 smithi017 ceph-mon[29862]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:24.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:24 smithi017 ceph-mon[29862]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:19:24.802 INFO:teuthology.orchestra.run.smithi001.stdout:55834574863 2024-03-11T21:19:25.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:25 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/1887695250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:19:25.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:25 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2903500401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:19:25.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:25 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:25.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:25 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:25.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:25 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/1887695250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:19:25.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:25 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2903500401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:19:25.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:25 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:25.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:25 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:25.642 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574862 got 55834574863 for osd.1 2024-03-11T21:19:25.642 DEBUG:teuthology.parallel:result is None 2024-03-11T21:19:25.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:25 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/1887695250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:19:25.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:25 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2903500401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:19:25.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:25 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:25.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:25 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:26.205 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:26 smithi001 ceph-mon[25774]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:19:26.289 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411337 got 77309411338 for osd.2 2024-03-11T21:19:26.289 DEBUG:teuthology.parallel:result is None 2024-03-11T21:19:26.552 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:26 smithi204 ceph-mon[30291]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:19:26.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:26 smithi017 ceph-mon[29862]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:19:27.096 INFO:teuthology.orchestra.run.smithi001.stdout:34359738390 2024-03-11T21:19:27.578 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.578 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: Adjusting osd_memory_target on smithi001 to 16069M 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: Updating smithi001:/etc/ceph/ceph.conf 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-11T21:19:27.579 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:27 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/2348160960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: Adjusting osd_memory_target on smithi001 to 16069M 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:27.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:27.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: Updating smithi001:/etc/ceph/ceph.conf 2024-03-11T21:19:27.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-11T21:19:27.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-11T21:19:27.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:27 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/2348160960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: Adjusting osd_memory_target on smithi001 to 16069M 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:27.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: Updating smithi001:/etc/ceph/ceph.conf 2024-03-11T21:19:27.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-11T21:19:27.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: Updating smithi204:/etc/ceph/ceph.conf 2024-03-11T21:19:27.803 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:27 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/2348160960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:19:27.898 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738388 got 34359738390 for osd.0 2024-03-11T21:19:27.899 DEBUG:teuthology.parallel:result is None 2024-03-11T21:19:27.899 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:19:27.899 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:19:28.146 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:28.532 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.532 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.532 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.534 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-mon[25774]: Deploying daemon alertmanager.a on smithi001 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.676 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:28 smithi017 ceph-mon[29862]: Deploying daemon alertmanager.a on smithi001 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.703 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Updating smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:28.704 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:28 smithi204 ceph-mon[30291]: Deploying daemon alertmanager.a on smithi001 2024-03-11T21:19:28.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:28 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ::ffff:172.21.15.17 - - [11/Mar/2024:21:19:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:30.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:30 smithi001 ceph-mon[25774]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:19:30.674 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:30 smithi017 ceph-mon[29862]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:19:30.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:19:30.690 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-11T21:19:30.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:30 smithi204 ceph-mon[30291]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:19:31.969 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-11T21:19:30.686880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595984,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.002432"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674270+0000","last_change":"2024-03-11T21:18:54.268801+0000","last_active":"2024-03-11T21:19:18.674270+0000","last_peered":"2024-03-11T21:19:18.674270+0000","last_clean":"2024-03-11T21:19:18.674270+0000","last_became_active":"2024-03-11T21:18:54.267796+0000","last_became_peered":"2024-03-11T21:18:54.267796+0000","last_unstale":"2024-03-11T21:19:18.674270+0000","last_undegraded":"2024-03-11T21:19:18.674270+0000","last_fullsized":"2024-03-11T21:19:18.674270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:25:59.917624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673557+0000","last_change":"2024-03-11T21:18:52.259801+0000","last_active":"2024-03-11T21:19:18.673557+0000","last_peered":"2024-03-11T21:19:18.673557+0000","last_clean":"2024-03-11T21:19:18.673557+0000","last_became_active":"2024-03-11T21:18:52.259449+0000","last_became_peered":"2024-03-11T21:18:52.259449+0000","last_unstale":"2024-03-11T21:19:18.673557+0000","last_undegraded":"2024-03-11T21:19:18.673557+0000","last_fullsized":"2024-03-11T21:19:18.673557+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:18:18.356017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677943+0000","last_change":"2024-03-11T21:18:48.244934+0000","last_active":"2024-03-11T21:19:18.677943+0000","last_peered":"2024-03-11T21:19:18.677943+0000","last_clean":"2024-03-11T21:19:18.677943+0000","last_became_active":"2024-03-11T21:18:48.243621+0000","last_became_peered":"2024-03-11T21:18:48.243621+0000","last_unstale":"2024-03-11T21:19:18.677943+0000","last_undegraded":"2024-03-11T21:19:18.677943+0000","last_fullsized":"2024-03-11T21:19:18.677943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:21:41.655490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676701+0000","last_change":"2024-03-11T21:18:50.258500+0000","last_active":"2024-03-11T21:19:18.676701+0000","last_peered":"2024-03-11T21:19:18.676701+0000","last_clean":"2024-03-11T21:19:18.676701+0000","last_became_active":"2024-03-11T21:18:50.257355+0000","last_became_peered":"2024-03-11T21:18:50.257355+0000","last_unstale":"2024-03-11T21:19:18.676701+0000","last_undegraded":"2024-03-11T21:19:18.676701+0000","last_fullsized":"2024-03-11T21:19:18.676701+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:43:14.949706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673225+0000","last_change":"2024-03-11T21:18:54.270030+0000","last_active":"2024-03-11T21:19:18.673225+0000","last_peered":"2024-03-11T21:19:18.673225+0000","last_clean":"2024-03-11T21:19:18.673225+0000","last_became_active":"2024-03-11T21:18:54.269284+0000","last_became_peered":"2024-03-11T21:18:54.269284+0000","last_unstale":"2024-03-11T21:19:18.673225+0000","last_undegraded":"2024-03-11T21:19:18.673225+0000","last_fullsized":"2024-03-11T21:19:18.673225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:33.322697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673782+0000","last_change":"2024-03-11T21:18:52.258854+0000","last_active":"2024-03-11T21:19:18.673782+0000","last_peered":"2024-03-11T21:19:18.673782+0000","last_clean":"2024-03-11T21:19:18.673782+0000","last_became_active":"2024-03-11T21:18:52.257924+0000","last_became_peered":"2024-03-11T21:18:52.257924+0000","last_unstale":"2024-03-11T21:19:18.673782+0000","last_undegraded":"2024-03-11T21:19:18.673782+0000","last_fullsized":"2024-03-11T21:19:18.673782+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:15.300210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677025+0000","last_change":"2024-03-11T21:18:48.245070+0000","last_active":"2024-03-11T21:19:18.677025+0000","last_peered":"2024-03-11T21:19:18.677025+0000","last_clean":"2024-03-11T21:19:18.677025+0000","last_became_active":"2024-03-11T21:18:48.243854+0000","last_became_peered":"2024-03-11T21:18:48.243854+0000","last_unstale":"2024-03-11T21:19:18.677025+0000","last_undegraded":"2024-03-11T21:19:18.677025+0000","last_fullsized":"2024-03-11T21:19:18.677025+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:22.577485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677555+0000","last_change":"2024-03-11T21:18:50.256291+0000","last_active":"2024-03-11T21:19:18.677555+0000","last_peered":"2024-03-11T21:19:18.677555+0000","last_clean":"2024-03-11T21:19:18.677555+0000","last_became_active":"2024-03-11T21:18:50.255871+0000","last_became_peered":"2024-03-11T21:18:50.255871+0000","last_unstale":"2024-03-11T21:19:18.677555+0000","last_undegraded":"2024-03-11T21:19:18.677555+0000","last_fullsized":"2024-03-11T21:19:18.677555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:15:10.582718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673760+0000","last_change":"2024-03-11T21:18:54.269077+0000","last_active":"2024-03-11T21:19:18.673760+0000","last_peered":"2024-03-11T21:19:18.673760+0000","last_clean":"2024-03-11T21:19:18.673760+0000","last_became_active":"2024-03-11T21:18:54.268687+0000","last_became_peered":"2024-03-11T21:18:54.268687+0000","last_unstale":"2024-03-11T21:19:18.673760+0000","last_undegraded":"2024-03-11T21:19:18.673760+0000","last_fullsized":"2024-03-11T21:19:18.673760+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:29.930763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674184+0000","last_change":"2024-03-11T21:18:52.257886+0000","last_active":"2024-03-11T21:19:18.674184+0000","last_peered":"2024-03-11T21:19:18.674184+0000","last_clean":"2024-03-11T21:19:18.674184+0000","last_became_active":"2024-03-11T21:18:52.257495+0000","last_became_peered":"2024-03-11T21:18:52.257495+0000","last_unstale":"2024-03-11T21:19:18.674184+0000","last_undegraded":"2024-03-11T21:19:18.674184+0000","last_fullsized":"2024-03-11T21:19:18.674184+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:43.742084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675804+0000","last_change":"2024-03-11T21:18:48.247988+0000","last_active":"2024-03-11T21:19:18.675804+0000","last_peered":"2024-03-11T21:19:18.675804+0000","last_clean":"2024-03-11T21:19:18.675804+0000","last_became_active":"2024-03-11T21:18:48.247716+0000","last_became_peered":"2024-03-11T21:18:48.247716+0000","last_unstale":"2024-03-11T21:19:18.675804+0000","last_undegraded":"2024-03-11T21:19:18.675804+0000","last_fullsized":"2024-03-11T21:19:18.675804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:52:43.612188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675321+0000","last_change":"2024-03-11T21:18:50.256296+0000","last_active":"2024-03-11T21:19:18.675321+0000","last_peered":"2024-03-11T21:19:18.675321+0000","last_clean":"2024-03-11T21:19:18.675321+0000","last_became_active":"2024-03-11T21:18:50.255951+0000","last_became_peered":"2024-03-11T21:18:50.255951+0000","last_unstale":"2024-03-11T21:19:18.675321+0000","last_undegraded":"2024-03-11T21:19:18.675321+0000","last_fullsized":"2024-03-11T21:19:18.675321+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:03.095698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674156+0000","last_change":"2024-03-11T21:18:54.270704+0000","last_active":"2024-03-11T21:19:18.674156+0000","last_peered":"2024-03-11T21:19:18.674156+0000","last_clean":"2024-03-11T21:19:18.674156+0000","last_became_active":"2024-03-11T21:18:54.269555+0000","last_became_peered":"2024-03-11T21:18:54.269555+0000","last_unstale":"2024-03-11T21:19:18.674156+0000","last_undegraded":"2024-03-11T21:19:18.674156+0000","last_fullsized":"2024-03-11T21:19:18.674156+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:51.587929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673536+0000","last_change":"2024-03-11T21:18:52.257020+0000","last_active":"2024-03-11T21:19:18.673536+0000","last_peered":"2024-03-11T21:19:18.673536+0000","last_clean":"2024-03-11T21:19:18.673536+0000","last_became_active":"2024-03-11T21:18:52.256248+0000","last_became_peered":"2024-03-11T21:18:52.256248+0000","last_unstale":"2024-03-11T21:19:18.673536+0000","last_undegraded":"2024-03-11T21:19:18.673536+0000","last_fullsized":"2024-03-11T21:19:18.673536+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:53.234541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676123+0000","last_change":"2024-03-11T21:18:48.247232+0000","last_active":"2024-03-11T21:19:18.676123+0000","last_peered":"2024-03-11T21:19:18.676123+0000","last_clean":"2024-03-11T21:19:18.676123+0000","last_became_active":"2024-03-11T21:18:48.246964+0000","last_became_peered":"2024-03-11T21:18:48.246964+0000","last_unstale":"2024-03-11T21:19:18.676123+0000","last_undegraded":"2024-03-11T21:19:18.676123+0000","last_fullsized":"2024-03-11T21:19:18.676123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:19.772442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675662+0000","last_change":"2024-03-11T21:18:50.255375+0000","last_active":"2024-03-11T21:19:18.675662+0000","last_peered":"2024-03-11T21:19:18.675662+0000","last_clean":"2024-03-11T21:19:18.675662+0000","last_became_active":"2024-03-11T21:18:50.255028+0000","last_became_peered":"2024-03-11T21:18:50.255028+0000","last_unstale":"2024-03-11T21:19:18.675662+0000","last_undegraded":"2024-03-11T21:19:18.675662+0000","last_fullsized":"2024-03-11T21:19:18.675662+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:51:54.280374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674307+0000","last_change":"2024-03-11T21:18:54.270647+0000","last_active":"2024-03-11T21:19:18.674307+0000","last_peered":"2024-03-11T21:19:18.674307+0000","last_clean":"2024-03-11T21:19:18.674307+0000","last_became_active":"2024-03-11T21:18:54.270206+0000","last_became_peered":"2024-03-11T21:18:54.270206+0000","last_unstale":"2024-03-11T21:19:18.674307+0000","last_undegraded":"2024-03-11T21:19:18.674307+0000","last_fullsized":"2024-03-11T21:19:18.674307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:23.514703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675133+0000","last_change":"2024-03-11T21:18:52.262335+0000","last_active":"2024-03-11T21:19:18.675133+0000","last_peered":"2024-03-11T21:19:18.675133+0000","last_clean":"2024-03-11T21:19:18.675133+0000","last_became_active":"2024-03-11T21:18:52.261325+0000","last_became_peered":"2024-03-11T21:18:52.261325+0000","last_unstale":"2024-03-11T21:19:18.675133+0000","last_undegraded":"2024-03-11T21:19:18.675133+0000","last_fullsized":"2024-03-11T21:19:18.675133+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:13:26.819492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675245+0000","last_change":"2024-03-11T21:18:48.247116+0000","last_active":"2024-03-11T21:19:18.675245+0000","last_peered":"2024-03-11T21:19:18.675245+0000","last_clean":"2024-03-11T21:19:18.675245+0000","last_became_active":"2024-03-11T21:18:48.246781+0000","last_became_peered":"2024-03-11T21:18:48.246781+0000","last_unstale":"2024-03-11T21:19:18.675245+0000","last_undegraded":"2024-03-11T21:19:18.675245+0000","last_fullsized":"2024-03-11T21:19:18.675245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:05.229493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674678+0000","last_change":"2024-03-11T21:18:50.255114+0000","last_active":"2024-03-11T21:19:18.674678+0000","last_peered":"2024-03-11T21:19:18.674678+0000","last_clean":"2024-03-11T21:19:18.674678+0000","last_became_active":"2024-03-11T21:18:50.254811+0000","last_became_peered":"2024-03-11T21:18:50.254811+0000","last_unstale":"2024-03-11T21:19:18.674678+0000","last_undegraded":"2024-03-11T21:19:18.674678+0000","last_fullsized":"2024-03-11T21:19:18.674678+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:46.755255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675788+0000","last_change":"2024-03-11T21:18:54.270901+0000","last_active":"2024-03-11T21:19:18.675788+0000","last_peered":"2024-03-11T21:19:18.675788+0000","last_clean":"2024-03-11T21:19:18.675788+0000","last_became_active":"2024-03-11T21:18:54.270175+0000","last_became_peered":"2024-03-11T21:18:54.270175+0000","last_unstale":"2024-03-11T21:19:18.675788+0000","last_undegraded":"2024-03-11T21:19:18.675788+0000","last_fullsized":"2024-03-11T21:19:18.675788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:35:19.940205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676018+0000","last_change":"2024-03-11T21:18:52.261762+0000","last_active":"2024-03-11T21:19:18.676018+0000","last_peered":"2024-03-11T21:19:18.676018+0000","last_clean":"2024-03-11T21:19:18.676018+0000","last_became_active":"2024-03-11T21:18:52.261101+0000","last_became_peered":"2024-03-11T21:18:52.261101+0000","last_unstale":"2024-03-11T21:19:18.676018+0000","last_undegraded":"2024-03-11T21:19:18.676018+0000","last_fullsized":"2024-03-11T21:19:18.676018+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:48.132420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674179+0000","last_change":"2024-03-11T21:18:48.247627+0000","last_active":"2024-03-11T21:19:18.674179+0000","last_peered":"2024-03-11T21:19:18.674179+0000","last_clean":"2024-03-11T21:19:18.674179+0000","last_became_active":"2024-03-11T21:18:48.247098+0000","last_became_peered":"2024-03-11T21:18:48.247098+0000","last_unstale":"2024-03-11T21:19:18.674179+0000","last_undegraded":"2024-03-11T21:19:18.674179+0000","last_fullsized":"2024-03-11T21:19:18.674179+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:32:42.106334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675079+0000","last_change":"2024-03-11T21:18:50.256293+0000","last_active":"2024-03-11T21:19:18.675079+0000","last_peered":"2024-03-11T21:19:18.675079+0000","last_clean":"2024-03-11T21:19:18.675079+0000","last_became_active":"2024-03-11T21:18:50.255976+0000","last_became_peered":"2024-03-11T21:18:50.255976+0000","last_unstale":"2024-03-11T21:19:18.675079+0000","last_undegraded":"2024-03-11T21:19:18.675079+0000","last_fullsized":"2024-03-11T21:19:18.675079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.787829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676180+0000","last_change":"2024-03-11T21:18:54.271422+0000","last_active":"2024-03-11T21:19:18.676180+0000","last_peered":"2024-03-11T21:19:18.676180+0000","last_clean":"2024-03-11T21:19:18.676180+0000","last_became_active":"2024-03-11T21:18:54.271108+0000","last_became_peered":"2024-03-11T21:18:54.271108+0000","last_unstale":"2024-03-11T21:19:18.676180+0000","last_undegraded":"2024-03-11T21:19:18.676180+0000","last_fullsized":"2024-03-11T21:19:18.676180+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:45:41.196419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295562+0000","last_change":"2024-03-11T21:18:52.262431+0000","last_active":"2024-03-11T21:19:23.295562+0000","last_peered":"2024-03-11T21:19:23.295562+0000","last_clean":"2024-03-11T21:19:23.295562+0000","last_became_active":"2024-03-11T21:18:52.262116+0000","last_became_peered":"2024-03-11T21:18:52.262116+0000","last_unstale":"2024-03-11T21:19:23.295562+0000","last_undegraded":"2024-03-11T21:19:23.295562+0000","last_fullsized":"2024-03-11T21:19:23.295562+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:22:34.117731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673445+0000","last_change":"2024-03-11T21:18:48.244992+0000","last_active":"2024-03-11T21:19:18.673445+0000","last_peered":"2024-03-11T21:19:18.673445+0000","last_clean":"2024-03-11T21:19:18.673445+0000","last_became_active":"2024-03-11T21:18:48.243766+0000","last_became_peered":"2024-03-11T21:18:48.243766+0000","last_unstale":"2024-03-11T21:19:18.673445+0000","last_undegraded":"2024-03-11T21:19:18.673445+0000","last_fullsized":"2024-03-11T21:19:18.673445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:09.893389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672551+0000","last_change":"2024-03-11T21:18:50.256193+0000","last_active":"2024-03-11T21:19:18.672551+0000","last_peered":"2024-03-11T21:19:18.672551+0000","last_clean":"2024-03-11T21:19:18.672551+0000","last_became_active":"2024-03-11T21:18:50.253709+0000","last_became_peered":"2024-03-11T21:18:50.253709+0000","last_unstale":"2024-03-11T21:19:18.672551+0000","last_undegraded":"2024-03-11T21:19:18.672551+0000","last_fullsized":"2024-03-11T21:19:18.672551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:25.617190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674638+0000","last_change":"2024-03-11T21:18:54.272403+0000","last_active":"2024-03-11T21:19:18.674638+0000","last_peered":"2024-03-11T21:19:18.674638+0000","last_clean":"2024-03-11T21:19:18.674638+0000","last_became_active":"2024-03-11T21:18:54.271448+0000","last_became_peered":"2024-03-11T21:18:54.271448+0000","last_unstale":"2024-03-11T21:19:18.674638+0000","last_undegraded":"2024-03-11T21:19:18.674638+0000","last_fullsized":"2024-03-11T21:19:18.674638+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:30:03.377329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676347+0000","last_change":"2024-03-11T21:18:52.263648+0000","last_active":"2024-03-11T21:19:18.676347+0000","last_peered":"2024-03-11T21:19:18.676347+0000","last_clean":"2024-03-11T21:19:18.676347+0000","last_became_active":"2024-03-11T21:18:52.263265+0000","last_became_peered":"2024-03-11T21:18:52.263265+0000","last_unstale":"2024-03-11T21:19:18.676347+0000","last_undegraded":"2024-03-11T21:19:18.676347+0000","last_fullsized":"2024-03-11T21:19:18.676347+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:36:35.868987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673636+0000","last_change":"2024-03-11T21:18:48.248405+0000","last_active":"2024-03-11T21:19:18.673636+0000","last_peered":"2024-03-11T21:19:18.673636+0000","last_clean":"2024-03-11T21:19:18.673636+0000","last_became_active":"2024-03-11T21:18:48.248131+0000","last_became_peered":"2024-03-11T21:18:48.248131+0000","last_unstale":"2024-03-11T21:19:18.673636+0000","last_undegraded":"2024-03-11T21:19:18.673636+0000","last_fullsized":"2024-03-11T21:19:18.673636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:23:54.755461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672890+0000","last_change":"2024-03-11T21:18:50.253219+0000","last_active":"2024-03-11T21:19:18.672890+0000","last_peered":"2024-03-11T21:19:18.672890+0000","last_clean":"2024-03-11T21:19:18.672890+0000","last_became_active":"2024-03-11T21:18:50.252031+0000","last_became_peered":"2024-03-11T21:18:50.252031+0000","last_unstale":"2024-03-11T21:19:18.672890+0000","last_undegraded":"2024-03-11T21:19:18.672890+0000","last_fullsized":"2024-03-11T21:19:18.672890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:48:04.913293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674460+0000","last_change":"2024-03-11T21:18:54.271299+0000","last_active":"2024-03-11T21:19:18.674460+0000","last_peered":"2024-03-11T21:19:18.674460+0000","last_clean":"2024-03-11T21:19:18.674460+0000","last_became_active":"2024-03-11T21:18:54.270919+0000","last_became_peered":"2024-03-11T21:18:54.270919+0000","last_unstale":"2024-03-11T21:19:18.674460+0000","last_undegraded":"2024-03-11T21:19:18.674460+0000","last_fullsized":"2024-03-11T21:19:18.674460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:54:46.145276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672713+0000","last_change":"2024-03-11T21:18:52.258529+0000","last_active":"2024-03-11T21:19:18.672713+0000","last_peered":"2024-03-11T21:19:18.672713+0000","last_clean":"2024-03-11T21:19:18.672713+0000","last_became_active":"2024-03-11T21:18:52.257868+0000","last_became_peered":"2024-03-11T21:18:52.257868+0000","last_unstale":"2024-03-11T21:19:18.672713+0000","last_undegraded":"2024-03-11T21:19:18.672713+0000","last_fullsized":"2024-03-11T21:19:18.672713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:04.556993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676091+0000","last_change":"2024-03-11T21:18:48.248882+0000","last_active":"2024-03-11T21:19:18.676091+0000","last_peered":"2024-03-11T21:19:18.676091+0000","last_clean":"2024-03-11T21:19:18.676091+0000","last_became_active":"2024-03-11T21:18:48.248283+0000","last_became_peered":"2024-03-11T21:18:48.248283+0000","last_unstale":"2024-03-11T21:19:18.676091+0000","last_undegraded":"2024-03-11T21:19:18.676091+0000","last_fullsized":"2024-03-11T21:19:18.676091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:16.551148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675950+0000","last_change":"2024-03-11T21:18:50.254073+0000","last_active":"2024-03-11T21:19:18.675950+0000","last_peered":"2024-03-11T21:19:18.675950+0000","last_clean":"2024-03-11T21:19:18.675950+0000","last_became_active":"2024-03-11T21:18:50.253420+0000","last_became_peered":"2024-03-11T21:18:50.253420+0000","last_unstale":"2024-03-11T21:19:18.675950+0000","last_undegraded":"2024-03-11T21:19:18.675950+0000","last_fullsized":"2024-03-11T21:19:18.675950+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:51.329348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673920+0000","last_change":"2024-03-11T21:18:54.270785+0000","last_active":"2024-03-11T21:19:18.673920+0000","last_peered":"2024-03-11T21:19:18.673920+0000","last_clean":"2024-03-11T21:19:18.673920+0000","last_became_active":"2024-03-11T21:18:54.270118+0000","last_became_peered":"2024-03-11T21:18:54.270118+0000","last_unstale":"2024-03-11T21:19:18.673920+0000","last_undegraded":"2024-03-11T21:19:18.673920+0000","last_fullsized":"2024-03-11T21:19:18.673920+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:48.827359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672889+0000","last_change":"2024-03-11T21:18:52.258813+0000","last_active":"2024-03-11T21:19:18.672889+0000","last_peered":"2024-03-11T21:19:18.672889+0000","last_clean":"2024-03-11T21:19:18.672889+0000","last_became_active":"2024-03-11T21:18:52.258463+0000","last_became_peered":"2024-03-11T21:18:52.258463+0000","last_unstale":"2024-03-11T21:19:18.672889+0000","last_undegraded":"2024-03-11T21:19:18.672889+0000","last_fullsized":"2024-03-11T21:19:18.672889+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:23:11.275587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675738+0000","last_change":"2024-03-11T21:18:48.248497+0000","last_active":"2024-03-11T21:19:18.675738+0000","last_peered":"2024-03-11T21:19:18.675738+0000","last_clean":"2024-03-11T21:19:18.675738+0000","last_became_active":"2024-03-11T21:18:48.248121+0000","last_became_peered":"2024-03-11T21:18:48.248121+0000","last_unstale":"2024-03-11T21:19:18.675738+0000","last_undegraded":"2024-03-11T21:19:18.675738+0000","last_fullsized":"2024-03-11T21:19:18.675738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:01.527285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677731+0000","last_change":"2024-03-11T21:18:50.256569+0000","last_active":"2024-03-11T21:19:18.677731+0000","last_peered":"2024-03-11T21:19:18.677731+0000","last_clean":"2024-03-11T21:19:18.677731+0000","last_became_active":"2024-03-11T21:18:50.256060+0000","last_became_peered":"2024-03-11T21:18:50.256060+0000","last_unstale":"2024-03-11T21:19:18.677731+0000","last_undegraded":"2024-03-11T21:19:18.677731+0000","last_fullsized":"2024-03-11T21:19:18.677731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:49:08.755492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674773+0000","last_change":"2024-03-11T21:18:54.268729+0000","last_active":"2024-03-11T21:19:18.674773+0000","last_peered":"2024-03-11T21:19:18.674773+0000","last_clean":"2024-03-11T21:19:18.674773+0000","last_became_active":"2024-03-11T21:18:54.268431+0000","last_became_peered":"2024-03-11T21:18:54.268431+0000","last_unstale":"2024-03-11T21:19:18.674773+0000","last_undegraded":"2024-03-11T21:19:18.674773+0000","last_fullsized":"2024-03-11T21:19:18.674773+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:42:23.684054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674300+0000","last_change":"2024-03-11T21:18:52.262140+0000","last_active":"2024-03-11T21:19:18.674300+0000","last_peered":"2024-03-11T21:19:18.674300+0000","last_clean":"2024-03-11T21:19:18.674300+0000","last_became_active":"2024-03-11T21:18:52.261854+0000","last_became_peered":"2024-03-11T21:18:52.261854+0000","last_unstale":"2024-03-11T21:19:18.674300+0000","last_undegraded":"2024-03-11T21:19:18.674300+0000","last_fullsized":"2024-03-11T21:19:18.674300+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:13.199921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675658+0000","last_change":"2024-03-11T21:18:48.250770+0000","last_active":"2024-03-11T21:19:18.675658+0000","last_peered":"2024-03-11T21:19:18.675658+0000","last_clean":"2024-03-11T21:19:18.675658+0000","last_became_active":"2024-03-11T21:18:48.250240+0000","last_became_peered":"2024-03-11T21:18:48.250240+0000","last_unstale":"2024-03-11T21:19:18.675658+0000","last_undegraded":"2024-03-11T21:19:18.675658+0000","last_fullsized":"2024-03-11T21:19:18.675658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:27:40.316142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676705+0000","last_change":"2024-03-11T21:18:50.254249+0000","last_active":"2024-03-11T21:19:18.676705+0000","last_peered":"2024-03-11T21:19:18.676705+0000","last_clean":"2024-03-11T21:19:18.676705+0000","last_became_active":"2024-03-11T21:18:50.253381+0000","last_became_peered":"2024-03-11T21:18:50.253381+0000","last_unstale":"2024-03-11T21:19:18.676705+0000","last_undegraded":"2024-03-11T21:19:18.676705+0000","last_fullsized":"2024-03-11T21:19:18.676705+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:10.091469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674232+0000","last_change":"2024-03-11T21:18:54.270313+0000","last_active":"2024-03-11T21:19:18.674232+0000","last_peered":"2024-03-11T21:19:18.674232+0000","last_clean":"2024-03-11T21:19:18.674232+0000","last_became_active":"2024-03-11T21:18:54.270016+0000","last_became_peered":"2024-03-11T21:18:54.270016+0000","last_unstale":"2024-03-11T21:19:18.674232+0000","last_undegraded":"2024-03-11T21:19:18.674232+0000","last_fullsized":"2024-03-11T21:19:18.674232+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:11:30.613393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673962+0000","last_change":"2024-03-11T21:18:52.260631+0000","last_active":"2024-03-11T21:19:18.673962+0000","last_peered":"2024-03-11T21:19:18.673962+0000","last_clean":"2024-03-11T21:19:18.673962+0000","last_became_active":"2024-03-11T21:18:52.260152+0000","last_became_peered":"2024-03-11T21:18:52.260152+0000","last_unstale":"2024-03-11T21:19:18.673962+0000","last_undegraded":"2024-03-11T21:19:18.673962+0000","last_fullsized":"2024-03-11T21:19:18.673962+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:45.743693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674684+0000","last_change":"2024-03-11T21:18:48.248071+0000","last_active":"2024-03-11T21:19:18.674684+0000","last_peered":"2024-03-11T21:19:18.674684+0000","last_clean":"2024-03-11T21:19:18.674684+0000","last_became_active":"2024-03-11T21:18:48.247186+0000","last_became_peered":"2024-03-11T21:18:48.247186+0000","last_unstale":"2024-03-11T21:19:18.674684+0000","last_undegraded":"2024-03-11T21:19:18.674684+0000","last_fullsized":"2024-03-11T21:19:18.674684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:13.309334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675463+0000","last_change":"2024-03-11T21:18:50.254918+0000","last_active":"2024-03-11T21:19:18.675463+0000","last_peered":"2024-03-11T21:19:18.675463+0000","last_clean":"2024-03-11T21:19:18.675463+0000","last_became_active":"2024-03-11T21:18:50.254649+0000","last_became_peered":"2024-03-11T21:18:50.254649+0000","last_unstale":"2024-03-11T21:19:18.675463+0000","last_undegraded":"2024-03-11T21:19:18.675463+0000","last_fullsized":"2024-03-11T21:19:18.675463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:14.847590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675293+0000","last_change":"2024-03-11T21:18:54.269071+0000","last_active":"2024-03-11T21:19:18.675293+0000","last_peered":"2024-03-11T21:19:18.675293+0000","last_clean":"2024-03-11T21:19:18.675293+0000","last_became_active":"2024-03-11T21:18:54.268743+0000","last_became_peered":"2024-03-11T21:18:54.268743+0000","last_unstale":"2024-03-11T21:19:18.675293+0000","last_undegraded":"2024-03-11T21:19:18.675293+0000","last_fullsized":"2024-03-11T21:19:18.675293+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:41.549264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295624+0000","last_change":"2024-03-11T21:18:52.262178+0000","last_active":"2024-03-11T21:19:23.295624+0000","last_peered":"2024-03-11T21:19:23.295624+0000","last_clean":"2024-03-11T21:19:23.295624+0000","last_became_active":"2024-03-11T21:18:52.260971+0000","last_became_peered":"2024-03-11T21:18:52.260971+0000","last_unstale":"2024-03-11T21:19:23.295624+0000","last_undegraded":"2024-03-11T21:19:23.295624+0000","last_fullsized":"2024-03-11T21:19:23.295624+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:38.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674889+0000","last_change":"2024-03-11T21:18:48.250477+0000","last_active":"2024-03-11T21:19:18.674889+0000","last_peered":"2024-03-11T21:19:18.674889+0000","last_clean":"2024-03-11T21:19:18.674889+0000","last_became_active":"2024-03-11T21:18:48.250195+0000","last_became_peered":"2024-03-11T21:18:48.250195+0000","last_unstale":"2024-03-11T21:19:18.674889+0000","last_undegraded":"2024-03-11T21:19:18.674889+0000","last_fullsized":"2024-03-11T21:19:18.674889+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:12:27.924310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674320+0000","last_change":"2024-03-11T21:18:50.252342+0000","last_active":"2024-03-11T21:19:18.674320+0000","last_peered":"2024-03-11T21:19:18.674320+0000","last_clean":"2024-03-11T21:19:18.674320+0000","last_became_active":"2024-03-11T21:18:50.251953+0000","last_became_peered":"2024-03-11T21:18:50.251953+0000","last_unstale":"2024-03-11T21:19:18.674320+0000","last_undegraded":"2024-03-11T21:19:18.674320+0000","last_fullsized":"2024-03-11T21:19:18.674320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:50.910861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674675+0000","last_change":"2024-03-11T21:18:54.271176+0000","last_active":"2024-03-11T21:19:18.674675+0000","last_peered":"2024-03-11T21:19:18.674675+0000","last_clean":"2024-03-11T21:19:18.674675+0000","last_became_active":"2024-03-11T21:18:54.269108+0000","last_became_peered":"2024-03-11T21:18:54.269108+0000","last_unstale":"2024-03-11T21:19:18.674675+0000","last_undegraded":"2024-03-11T21:19:18.674675+0000","last_fullsized":"2024-03-11T21:19:18.674675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:16:06.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295656+0000","last_change":"2024-03-11T21:18:52.261164+0000","last_active":"2024-03-11T21:19:23.295656+0000","last_peered":"2024-03-11T21:19:23.295656+0000","last_clean":"2024-03-11T21:19:23.295656+0000","last_became_active":"2024-03-11T21:18:52.260775+0000","last_became_peered":"2024-03-11T21:18:52.260775+0000","last_unstale":"2024-03-11T21:19:23.295656+0000","last_undegraded":"2024-03-11T21:19:23.295656+0000","last_fullsized":"2024-03-11T21:19:23.295656+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:46.520752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674224+0000","last_change":"2024-03-11T21:18:48.247699+0000","last_active":"2024-03-11T21:19:18.674224+0000","last_peered":"2024-03-11T21:19:18.674224+0000","last_clean":"2024-03-11T21:19:18.674224+0000","last_became_active":"2024-03-11T21:18:48.246931+0000","last_became_peered":"2024-03-11T21:18:48.246931+0000","last_unstale":"2024-03-11T21:19:18.674224+0000","last_undegraded":"2024-03-11T21:19:18.674224+0000","last_fullsized":"2024-03-11T21:19:18.674224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:12:46.448806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675310+0000","last_change":"2024-03-11T21:18:50.253989+0000","last_active":"2024-03-11T21:19:18.675310+0000","last_peered":"2024-03-11T21:19:18.675310+0000","last_clean":"2024-03-11T21:19:18.675310+0000","last_became_active":"2024-03-11T21:18:50.253522+0000","last_became_peered":"2024-03-11T21:18:50.253522+0000","last_unstale":"2024-03-11T21:19:18.675310+0000","last_undegraded":"2024-03-11T21:19:18.675310+0000","last_fullsized":"2024-03-11T21:19:18.675310+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:10.332041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674518+0000","last_change":"2024-03-11T21:18:54.275212+0000","last_active":"2024-03-11T21:19:18.674518+0000","last_peered":"2024-03-11T21:19:18.674518+0000","last_clean":"2024-03-11T21:19:18.674518+0000","last_became_active":"2024-03-11T21:18:54.274808+0000","last_became_peered":"2024-03-11T21:18:54.274808+0000","last_unstale":"2024-03-11T21:19:18.674518+0000","last_undegraded":"2024-03-11T21:19:18.674518+0000","last_fullsized":"2024-03-11T21:19:18.674518+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:04.848457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677472+0000","last_change":"2024-03-11T21:18:52.263762+0000","last_active":"2024-03-11T21:19:18.677472+0000","last_peered":"2024-03-11T21:19:18.677472+0000","last_clean":"2024-03-11T21:19:18.677472+0000","last_became_active":"2024-03-11T21:18:52.262815+0000","last_became_peered":"2024-03-11T21:18:52.262815+0000","last_unstale":"2024-03-11T21:19:18.677472+0000","last_undegraded":"2024-03-11T21:19:18.677472+0000","last_fullsized":"2024-03-11T21:19:18.677472+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:10:20.161132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673730+0000","last_change":"2024-03-11T21:18:48.248664+0000","last_active":"2024-03-11T21:19:18.673730+0000","last_peered":"2024-03-11T21:19:18.673730+0000","last_clean":"2024-03-11T21:19:18.673730+0000","last_became_active":"2024-03-11T21:18:48.247297+0000","last_became_peered":"2024-03-11T21:18:48.247297+0000","last_unstale":"2024-03-11T21:19:18.673730+0000","last_undegraded":"2024-03-11T21:19:18.673730+0000","last_fullsized":"2024-03-11T21:19:18.673730+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:19:11.465458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673350+0000","last_change":"2024-03-11T21:18:50.255500+0000","last_active":"2024-03-11T21:19:18.673350+0000","last_peered":"2024-03-11T21:19:18.673350+0000","last_clean":"2024-03-11T21:19:18.673350+0000","last_became_active":"2024-03-11T21:18:50.255157+0000","last_became_peered":"2024-03-11T21:18:50.255157+0000","last_unstale":"2024-03-11T21:19:18.673350+0000","last_undegraded":"2024-03-11T21:19:18.673350+0000","last_fullsized":"2024-03-11T21:19:18.673350+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:43.208883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677329+0000","last_change":"2024-03-11T21:18:54.271440+0000","last_active":"2024-03-11T21:19:18.677329+0000","last_peered":"2024-03-11T21:19:18.677329+0000","last_clean":"2024-03-11T21:19:18.677329+0000","last_became_active":"2024-03-11T21:18:54.271145+0000","last_became_peered":"2024-03-11T21:18:54.271145+0000","last_unstale":"2024-03-11T21:19:18.677329+0000","last_undegraded":"2024-03-11T21:19:18.677329+0000","last_fullsized":"2024-03-11T21:19:18.677329+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:58:10.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676434+0000","last_change":"2024-03-11T21:18:52.263749+0000","last_active":"2024-03-11T21:19:18.676434+0000","last_peered":"2024-03-11T21:19:18.676434+0000","last_clean":"2024-03-11T21:19:18.676434+0000","last_became_active":"2024-03-11T21:18:52.263403+0000","last_became_peered":"2024-03-11T21:18:52.263403+0000","last_unstale":"2024-03-11T21:19:18.676434+0000","last_undegraded":"2024-03-11T21:19:18.676434+0000","last_fullsized":"2024-03-11T21:19:18.676434+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:56:59.967077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673651+0000","last_change":"2024-03-11T21:18:48.248996+0000","last_active":"2024-03-11T21:19:18.673651+0000","last_peered":"2024-03-11T21:19:18.673651+0000","last_clean":"2024-03-11T21:19:18.673651+0000","last_became_active":"2024-03-11T21:18:48.248697+0000","last_became_peered":"2024-03-11T21:18:48.248697+0000","last_unstale":"2024-03-11T21:19:18.673651+0000","last_undegraded":"2024-03-11T21:19:18.673651+0000","last_fullsized":"2024-03-11T21:19:18.673651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:29.290103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673856+0000","last_change":"2024-03-11T21:18:50.259514+0000","last_active":"2024-03-11T21:19:18.673856+0000","last_peered":"2024-03-11T21:19:18.673856+0000","last_clean":"2024-03-11T21:19:18.673856+0000","last_became_active":"2024-03-11T21:18:50.259040+0000","last_became_peered":"2024-03-11T21:18:50.259040+0000","last_unstale":"2024-03-11T21:19:18.673856+0000","last_undegraded":"2024-03-11T21:19:18.673856+0000","last_fullsized":"2024-03-11T21:19:18.673856+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:51.913186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674549+0000","last_change":"2024-03-11T21:18:54.269923+0000","last_active":"2024-03-11T21:19:18.674549+0000","last_peered":"2024-03-11T21:19:18.674549+0000","last_clean":"2024-03-11T21:19:18.674549+0000","last_became_active":"2024-03-11T21:18:54.269569+0000","last_became_peered":"2024-03-11T21:18:54.269569+0000","last_unstale":"2024-03-11T21:19:18.674549+0000","last_undegraded":"2024-03-11T21:19:18.674549+0000","last_fullsized":"2024-03-11T21:19:18.674549+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:42:01.506214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676698+0000","last_change":"2024-03-11T21:18:48.247869+0000","last_active":"2024-03-11T21:19:18.676698+0000","last_peered":"2024-03-11T21:19:18.676698+0000","last_clean":"2024-03-11T21:19:18.676698+0000","last_became_active":"2024-03-11T21:18:48.247594+0000","last_became_peered":"2024-03-11T21:18:48.247594+0000","last_unstale":"2024-03-11T21:19:18.676698+0000","last_undegraded":"2024-03-11T21:19:18.676698+0000","last_fullsized":"2024-03-11T21:19:18.676698+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:57:53.797906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676522+0000","last_change":"2024-03-11T21:18:50.254338+0000","last_active":"2024-03-11T21:19:18.676522+0000","last_peered":"2024-03-11T21:19:18.676522+0000","last_clean":"2024-03-11T21:19:18.676522+0000","last_became_active":"2024-03-11T21:18:50.254086+0000","last_became_peered":"2024-03-11T21:18:50.254086+0000","last_unstale":"2024-03-11T21:19:18.676522+0000","last_undegraded":"2024-03-11T21:19:18.676522+0000","last_fullsized":"2024-03-11T21:19:18.676522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:30.941114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295680+0000","last_change":"2024-03-11T21:18:52.262882+0000","last_active":"2024-03-11T21:19:23.295680+0000","last_peered":"2024-03-11T21:19:23.295680+0000","last_clean":"2024-03-11T21:19:23.295680+0000","last_became_active":"2024-03-11T21:18:52.261840+0000","last_became_peered":"2024-03-11T21:18:52.261840+0000","last_unstale":"2024-03-11T21:19:23.295680+0000","last_undegraded":"2024-03-11T21:19:23.295680+0000","last_fullsized":"2024-03-11T21:19:23.295680+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:00.609082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672860+0000","last_change":"2024-03-11T21:18:54.267931+0000","last_active":"2024-03-11T21:19:18.672860+0000","last_peered":"2024-03-11T21:19:18.672860+0000","last_clean":"2024-03-11T21:19:18.672860+0000","last_became_active":"2024-03-11T21:18:54.267399+0000","last_became_peered":"2024-03-11T21:18:54.267399+0000","last_unstale":"2024-03-11T21:19:18.672860+0000","last_undegraded":"2024-03-11T21:19:18.672860+0000","last_fullsized":"2024-03-11T21:19:18.672860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:42:22.105023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675924+0000","last_change":"2024-03-11T21:18:48.245449+0000","last_active":"2024-03-11T21:19:18.675924+0000","last_peered":"2024-03-11T21:19:18.675924+0000","last_clean":"2024-03-11T21:19:18.675924+0000","last_became_active":"2024-03-11T21:18:48.245109+0000","last_became_peered":"2024-03-11T21:18:48.245109+0000","last_unstale":"2024-03-11T21:19:18.675924+0000","last_undegraded":"2024-03-11T21:19:18.675924+0000","last_fullsized":"2024-03-11T21:19:18.675924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:01.506661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295644+0000","last_change":"2024-03-11T21:18:50.255542+0000","last_active":"2024-03-11T21:19:23.295644+0000","last_peered":"2024-03-11T21:19:23.295644+0000","last_clean":"2024-03-11T21:19:23.295644+0000","last_became_active":"2024-03-11T21:18:50.255201+0000","last_became_peered":"2024-03-11T21:18:50.255201+0000","last_unstale":"2024-03-11T21:19:23.295644+0000","last_undegraded":"2024-03-11T21:19:23.295644+0000","last_fullsized":"2024-03-11T21:19:23.295644+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:12:45.241972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673990+0000","last_change":"2024-03-11T21:18:52.255738+0000","last_active":"2024-03-11T21:19:18.673990+0000","last_peered":"2024-03-11T21:19:18.673990+0000","last_clean":"2024-03-11T21:19:18.673990+0000","last_became_active":"2024-03-11T21:18:52.255512+0000","last_became_peered":"2024-03-11T21:18:52.255512+0000","last_unstale":"2024-03-11T21:19:18.673990+0000","last_undegraded":"2024-03-11T21:19:18.673990+0000","last_fullsized":"2024-03-11T21:19:18.673990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:56.558089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675068+0000","last_change":"2024-03-11T21:18:54.269672+0000","last_active":"2024-03-11T21:19:18.675068+0000","last_peered":"2024-03-11T21:19:18.675068+0000","last_clean":"2024-03-11T21:19:18.675068+0000","last_became_active":"2024-03-11T21:18:54.269345+0000","last_became_peered":"2024-03-11T21:18:54.269345+0000","last_unstale":"2024-03-11T21:19:18.675068+0000","last_undegraded":"2024-03-11T21:19:18.675068+0000","last_fullsized":"2024-03-11T21:19:18.675068+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:20:47.364834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674335+0000","last_change":"2024-03-11T21:18:48.247179+0000","last_active":"2024-03-11T21:19:18.674335+0000","last_peered":"2024-03-11T21:19:18.674335+0000","last_clean":"2024-03-11T21:19:18.674335+0000","last_became_active":"2024-03-11T21:18:48.246841+0000","last_became_peered":"2024-03-11T21:18:48.246841+0000","last_unstale":"2024-03-11T21:19:18.674335+0000","last_undegraded":"2024-03-11T21:19:18.674335+0000","last_fullsized":"2024-03-11T21:19:18.674335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:33:39.174507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675476+0000","last_change":"2024-03-11T21:18:50.253332+0000","last_active":"2024-03-11T21:19:18.675476+0000","last_peered":"2024-03-11T21:19:18.675476+0000","last_clean":"2024-03-11T21:19:18.675476+0000","last_became_active":"2024-03-11T21:18:50.252333+0000","last_became_peered":"2024-03-11T21:18:50.252333+0000","last_unstale":"2024-03-11T21:19:18.675476+0000","last_undegraded":"2024-03-11T21:19:18.675476+0000","last_fullsized":"2024-03-11T21:19:18.675476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:31.270474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675387+0000","last_change":"2024-03-11T21:18:52.263625+0000","last_active":"2024-03-11T21:19:18.675387+0000","last_peered":"2024-03-11T21:19:18.675387+0000","last_clean":"2024-03-11T21:19:18.675387+0000","last_became_active":"2024-03-11T21:18:52.263227+0000","last_became_peered":"2024-03-11T21:18:52.263227+0000","last_unstale":"2024-03-11T21:19:18.675387+0000","last_undegraded":"2024-03-11T21:19:18.675387+0000","last_fullsized":"2024-03-11T21:19:18.675387+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:04.484588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677170+0000","last_change":"2024-03-11T21:18:54.271572+0000","last_active":"2024-03-11T21:19:18.677170+0000","last_peered":"2024-03-11T21:19:18.677170+0000","last_clean":"2024-03-11T21:19:18.677170+0000","last_became_active":"2024-03-11T21:18:54.271240+0000","last_became_peered":"2024-03-11T21:18:54.271240+0000","last_unstale":"2024-03-11T21:19:18.677170+0000","last_undegraded":"2024-03-11T21:19:18.677170+0000","last_fullsized":"2024-03-11T21:19:18.677170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:22.422794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673399+0000","last_change":"2024-03-11T21:18:50.259525+0000","last_active":"2024-03-11T21:19:18.673399+0000","last_peered":"2024-03-11T21:19:18.673399+0000","last_clean":"2024-03-11T21:19:18.673399+0000","last_became_active":"2024-03-11T21:18:50.259248+0000","last_became_peered":"2024-03-11T21:18:50.259248+0000","last_unstale":"2024-03-11T21:19:18.673399+0000","last_undegraded":"2024-03-11T21:19:18.673399+0000","last_fullsized":"2024-03-11T21:19:18.673399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:08.756866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673125+0000","last_change":"2024-03-11T21:18:48.245861+0000","last_active":"2024-03-11T21:19:18.673125+0000","last_peered":"2024-03-11T21:19:18.673125+0000","last_clean":"2024-03-11T21:19:18.673125+0000","last_became_active":"2024-03-11T21:18:48.245351+0000","last_became_peered":"2024-03-11T21:18:48.245351+0000","last_unstale":"2024-03-11T21:19:18.673125+0000","last_undegraded":"2024-03-11T21:19:18.673125+0000","last_fullsized":"2024-03-11T21:19:18.673125+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:47:55.849112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677622+0000","last_change":"2024-03-11T21:18:52.264231+0000","last_active":"2024-03-11T21:19:18.677622+0000","last_peered":"2024-03-11T21:19:18.677622+0000","last_clean":"2024-03-11T21:19:18.677622+0000","last_became_active":"2024-03-11T21:18:52.263604+0000","last_became_peered":"2024-03-11T21:18:52.263604+0000","last_unstale":"2024-03-11T21:19:18.677622+0000","last_undegraded":"2024-03-11T21:19:18.677622+0000","last_fullsized":"2024-03-11T21:19:18.677622+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:37:01.641209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675483+0000","last_change":"2024-03-11T21:18:54.270112+0000","last_active":"2024-03-11T21:19:18.675483+0000","last_peered":"2024-03-11T21:19:18.675483+0000","last_clean":"2024-03-11T21:19:18.675483+0000","last_became_active":"2024-03-11T21:18:54.269793+0000","last_became_peered":"2024-03-11T21:18:54.269793+0000","last_unstale":"2024-03-11T21:19:18.675483+0000","last_undegraded":"2024-03-11T21:19:18.675483+0000","last_fullsized":"2024-03-11T21:19:18.675483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:36.132338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674120+0000","last_change":"2024-03-11T21:18:50.252671+0000","last_active":"2024-03-11T21:19:18.674120+0000","last_peered":"2024-03-11T21:19:18.674120+0000","last_clean":"2024-03-11T21:19:18.674120+0000","last_became_active":"2024-03-11T21:18:50.252279+0000","last_became_peered":"2024-03-11T21:18:50.252279+0000","last_unstale":"2024-03-11T21:19:18.674120+0000","last_undegraded":"2024-03-11T21:19:18.674120+0000","last_fullsized":"2024-03-11T21:19:18.674120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:47:01.954334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674416+0000","last_change":"2024-03-11T21:18:48.247627+0000","last_active":"2024-03-11T21:19:18.674416+0000","last_peered":"2024-03-11T21:19:18.674416+0000","last_clean":"2024-03-11T21:19:18.674416+0000","last_became_active":"2024-03-11T21:18:48.247095+0000","last_became_peered":"2024-03-11T21:18:48.247095+0000","last_unstale":"2024-03-11T21:19:18.674416+0000","last_undegraded":"2024-03-11T21:19:18.674416+0000","last_fullsized":"2024-03-11T21:19:18.674416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:07:13.692264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:20.769805+0000","last_change":"2024-03-11T21:18:37.228580+0000","last_active":"2024-03-11T21:19:20.769805+0000","last_peered":"2024-03-11T21:19:20.769805+0000","last_clean":"2024-03-11T21:19:20.769805+0000","last_became_active":"2024-03-11T21:18:37.228208+0000","last_became_peered":"2024-03-11T21:18:37.228208+0000","last_unstale":"2024-03-11T21:19:20.769805+0000","last_undegraded":"2024-03-11T21:19:20.769805+0000","last_fullsized":"2024-03-11T21:19:20.769805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:36.216238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:36.216238+0000","last_clean_scrub_stamp":"2024-03-11T21:18:36.216238+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:55.801174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675745+0000","last_change":"2024-03-11T21:18:52.257644+0000","last_active":"2024-03-11T21:19:18.675745+0000","last_peered":"2024-03-11T21:19:18.675745+0000","last_clean":"2024-03-11T21:19:18.675745+0000","last_became_active":"2024-03-11T21:18:52.257204+0000","last_became_peered":"2024-03-11T21:18:52.257204+0000","last_unstale":"2024-03-11T21:19:18.675745+0000","last_undegraded":"2024-03-11T21:19:18.675745+0000","last_fullsized":"2024-03-11T21:19:18.675745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:44:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674725+0000","last_change":"2024-03-11T21:18:54.272781+0000","last_active":"2024-03-11T21:19:18.674725+0000","last_peered":"2024-03-11T21:19:18.674725+0000","last_clean":"2024-03-11T21:19:18.674725+0000","last_became_active":"2024-03-11T21:18:54.272442+0000","last_became_peered":"2024-03-11T21:18:54.272442+0000","last_unstale":"2024-03-11T21:19:18.674725+0000","last_undegraded":"2024-03-11T21:19:18.674725+0000","last_fullsized":"2024-03-11T21:19:18.674725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:06:31.016510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673479+0000","last_change":"2024-03-11T21:18:50.256073+0000","last_active":"2024-03-11T21:19:18.673479+0000","last_peered":"2024-03-11T21:19:18.673479+0000","last_clean":"2024-03-11T21:19:18.673479+0000","last_became_active":"2024-03-11T21:18:50.253649+0000","last_became_peered":"2024-03-11T21:18:50.253649+0000","last_unstale":"2024-03-11T21:19:18.673479+0000","last_undegraded":"2024-03-11T21:19:18.673479+0000","last_fullsized":"2024-03-11T21:19:18.673479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:18:54.215008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673557+0000","last_change":"2024-03-11T21:18:48.250192+0000","last_active":"2024-03-11T21:19:18.673557+0000","last_peered":"2024-03-11T21:19:18.673557+0000","last_clean":"2024-03-11T21:19:18.673557+0000","last_became_active":"2024-03-11T21:18:48.249808+0000","last_became_peered":"2024-03-11T21:18:48.249808+0000","last_unstale":"2024-03-11T21:19:18.673557+0000","last_undegraded":"2024-03-11T21:19:18.673557+0000","last_fullsized":"2024-03-11T21:19:18.673557+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:33:00.890001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675640+0000","last_change":"2024-03-11T21:18:52.262568+0000","last_active":"2024-03-11T21:19:18.675640+0000","last_peered":"2024-03-11T21:19:18.675640+0000","last_clean":"2024-03-11T21:19:18.675640+0000","last_became_active":"2024-03-11T21:18:52.262262+0000","last_became_peered":"2024-03-11T21:18:52.262262+0000","last_unstale":"2024-03-11T21:19:18.675640+0000","last_undegraded":"2024-03-11T21:19:18.675640+0000","last_fullsized":"2024-03-11T21:19:18.675640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:05:19.605432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674073+0000","last_change":"2024-03-11T21:18:54.269037+0000","last_active":"2024-03-11T21:19:18.674073+0000","last_peered":"2024-03-11T21:19:18.674073+0000","last_clean":"2024-03-11T21:19:18.674073+0000","last_became_active":"2024-03-11T21:18:54.268275+0000","last_became_peered":"2024-03-11T21:18:54.268275+0000","last_unstale":"2024-03-11T21:19:18.674073+0000","last_undegraded":"2024-03-11T21:19:18.674073+0000","last_fullsized":"2024-03-11T21:19:18.674073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:42:01.189789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675554+0000","last_change":"2024-03-11T21:18:48.248663+0000","last_active":"2024-03-11T21:19:18.675554+0000","last_peered":"2024-03-11T21:19:18.675554+0000","last_clean":"2024-03-11T21:19:18.675554+0000","last_became_active":"2024-03-11T21:18:48.248376+0000","last_became_peered":"2024-03-11T21:18:48.248376+0000","last_unstale":"2024-03-11T21:19:18.675554+0000","last_undegraded":"2024-03-11T21:19:18.675554+0000","last_fullsized":"2024-03-11T21:19:18.675554+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:08:26.780594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":80,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295471+0000","last_change":"2024-03-11T21:18:50.253683+0000","last_active":"2024-03-11T21:19:23.295471+0000","last_peered":"2024-03-11T21:19:23.295471+0000","last_clean":"2024-03-11T21:19:23.295471+0000","last_became_active":"2024-03-11T21:18:50.253244+0000","last_became_peered":"2024-03-11T21:18:50.253244+0000","last_unstale":"2024-03-11T21:19:23.295471+0000","last_undegraded":"2024-03-11T21:19:23.295471+0000","last_fullsized":"2024-03-11T21:19:23.295471+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:46:42.254634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295639+0000","last_change":"2024-03-11T21:18:52.261066+0000","last_active":"2024-03-11T21:19:23.295639+0000","last_peered":"2024-03-11T21:19:23.295639+0000","last_clean":"2024-03-11T21:19:23.295639+0000","last_became_active":"2024-03-11T21:18:52.260664+0000","last_became_peered":"2024-03-11T21:18:52.260664+0000","last_unstale":"2024-03-11T21:19:23.295639+0000","last_undegraded":"2024-03-11T21:19:23.295639+0000","last_fullsized":"2024-03-11T21:19:23.295639+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:53:05.029022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673263+0000","last_change":"2024-03-11T21:18:54.271138+0000","last_active":"2024-03-11T21:19:18.673263+0000","last_peered":"2024-03-11T21:19:18.673263+0000","last_clean":"2024-03-11T21:19:18.673263+0000","last_became_active":"2024-03-11T21:18:54.270413+0000","last_became_peered":"2024-03-11T21:18:54.270413+0000","last_unstale":"2024-03-11T21:19:18.673263+0000","last_undegraded":"2024-03-11T21:19:18.673263+0000","last_fullsized":"2024-03-11T21:19:18.673263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:17.724950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677139+0000","last_change":"2024-03-11T21:18:48.246093+0000","last_active":"2024-03-11T21:19:18.677139+0000","last_peered":"2024-03-11T21:19:18.677139+0000","last_clean":"2024-03-11T21:19:18.677139+0000","last_became_active":"2024-03-11T21:18:48.245628+0000","last_became_peered":"2024-03-11T21:18:48.245628+0000","last_unstale":"2024-03-11T21:19:18.677139+0000","last_undegraded":"2024-03-11T21:19:18.677139+0000","last_fullsized":"2024-03-11T21:19:18.677139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:09:46.690889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.678030+0000","last_change":"2024-03-11T21:18:50.258139+0000","last_active":"2024-03-11T21:19:18.678030+0000","last_peered":"2024-03-11T21:19:18.678030+0000","last_clean":"2024-03-11T21:19:18.678030+0000","last_became_active":"2024-03-11T21:18:50.256723+0000","last_became_peered":"2024-03-11T21:18:50.256723+0000","last_unstale":"2024-03-11T21:19:18.678030+0000","last_undegraded":"2024-03-11T21:19:18.678030+0000","last_fullsized":"2024-03-11T21:19:18.678030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:32.240855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295462+0000","last_change":"2024-03-11T21:18:52.259011+0000","last_active":"2024-03-11T21:19:23.295462+0000","last_peered":"2024-03-11T21:19:23.295462+0000","last_clean":"2024-03-11T21:19:23.295462+0000","last_became_active":"2024-03-11T21:18:52.258629+0000","last_became_peered":"2024-03-11T21:18:52.258629+0000","last_unstale":"2024-03-11T21:19:23.295462+0000","last_undegraded":"2024-03-11T21:19:23.295462+0000","last_fullsized":"2024-03-11T21:19:23.295462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:53.726500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674492+0000","last_change":"2024-03-11T21:18:54.269645+0000","last_active":"2024-03-11T21:19:18.674492+0000","last_peered":"2024-03-11T21:19:18.674492+0000","last_clean":"2024-03-11T21:19:18.674492+0000","last_became_active":"2024-03-11T21:18:54.269306+0000","last_became_peered":"2024-03-11T21:18:54.269306+0000","last_unstale":"2024-03-11T21:19:18.674492+0000","last_undegraded":"2024-03-11T21:19:18.674492+0000","last_fullsized":"2024-03-11T21:19:18.674492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:56.987598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675813+0000","last_change":"2024-03-11T21:18:48.249851+0000","last_active":"2024-03-11T21:19:18.675813+0000","last_peered":"2024-03-11T21:19:18.675813+0000","last_clean":"2024-03-11T21:19:18.675813+0000","last_became_active":"2024-03-11T21:18:48.249511+0000","last_became_peered":"2024-03-11T21:18:48.249511+0000","last_unstale":"2024-03-11T21:19:18.675813+0000","last_undegraded":"2024-03-11T21:19:18.675813+0000","last_fullsized":"2024-03-11T21:19:18.675813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:12.080235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676576+0000","last_change":"2024-03-11T21:18:50.253791+0000","last_active":"2024-03-11T21:19:18.676576+0000","last_peered":"2024-03-11T21:19:18.676576+0000","last_clean":"2024-03-11T21:19:18.676576+0000","last_became_active":"2024-03-11T21:18:50.252999+0000","last_became_peered":"2024-03-11T21:18:50.252999+0000","last_unstale":"2024-03-11T21:19:18.676576+0000","last_undegraded":"2024-03-11T21:19:18.676576+0000","last_fullsized":"2024-03-11T21:19:18.676576+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:46.092445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673665+0000","last_change":"2024-03-11T21:18:52.262895+0000","last_active":"2024-03-11T21:19:18.673665+0000","last_peered":"2024-03-11T21:19:18.673665+0000","last_clean":"2024-03-11T21:19:18.673665+0000","last_became_active":"2024-03-11T21:18:52.261884+0000","last_became_peered":"2024-03-11T21:18:52.261884+0000","last_unstale":"2024-03-11T21:19:18.673665+0000","last_undegraded":"2024-03-11T21:19:18.673665+0000","last_fullsized":"2024-03-11T21:19:18.673665+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:01:24.568625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675402+0000","last_change":"2024-03-11T21:18:54.271306+0000","last_active":"2024-03-11T21:19:18.675402+0000","last_peered":"2024-03-11T21:19:18.675402+0000","last_clean":"2024-03-11T21:19:18.675402+0000","last_became_active":"2024-03-11T21:18:54.270132+0000","last_became_peered":"2024-03-11T21:18:54.270132+0000","last_unstale":"2024-03-11T21:19:18.675402+0000","last_undegraded":"2024-03-11T21:19:18.675402+0000","last_fullsized":"2024-03-11T21:19:18.675402+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:48.135906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295678+0000","last_change":"2024-03-11T21:18:52.258699+0000","last_active":"2024-03-11T21:19:23.295678+0000","last_peered":"2024-03-11T21:19:23.295678+0000","last_clean":"2024-03-11T21:19:23.295678+0000","last_became_active":"2024-03-11T21:18:52.257886+0000","last_became_peered":"2024-03-11T21:18:52.257886+0000","last_unstale":"2024-03-11T21:19:23.295678+0000","last_undegraded":"2024-03-11T21:19:23.295678+0000","last_fullsized":"2024-03-11T21:19:23.295678+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:09.010264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673571+0000","last_change":"2024-03-11T21:18:48.250174+0000","last_active":"2024-03-11T21:19:18.673571+0000","last_peered":"2024-03-11T21:19:18.673571+0000","last_clean":"2024-03-11T21:19:18.673571+0000","last_became_active":"2024-03-11T21:18:48.249814+0000","last_became_peered":"2024-03-11T21:18:48.249814+0000","last_unstale":"2024-03-11T21:19:18.673571+0000","last_undegraded":"2024-03-11T21:19:18.673571+0000","last_fullsized":"2024-03-11T21:19:18.673571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:08.168756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674077+0000","last_change":"2024-03-11T21:18:50.259280+0000","last_active":"2024-03-11T21:19:18.674077+0000","last_peered":"2024-03-11T21:19:18.674077+0000","last_clean":"2024-03-11T21:19:18.674077+0000","last_became_active":"2024-03-11T21:18:50.258990+0000","last_became_peered":"2024-03-11T21:18:50.258990+0000","last_unstale":"2024-03-11T21:19:18.674077+0000","last_undegraded":"2024-03-11T21:19:18.674077+0000","last_fullsized":"2024-03-11T21:19:18.674077+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:36:06.935178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675733+0000","last_change":"2024-03-11T21:18:54.272376+0000","last_active":"2024-03-11T21:19:18.675733+0000","last_peered":"2024-03-11T21:19:18.675733+0000","last_clean":"2024-03-11T21:19:18.675733+0000","last_became_active":"2024-03-11T21:18:54.272053+0000","last_became_peered":"2024-03-11T21:18:54.272053+0000","last_unstale":"2024-03-11T21:19:18.675733+0000","last_undegraded":"2024-03-11T21:19:18.675733+0000","last_fullsized":"2024-03-11T21:19:18.675733+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:34.884608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677518+0000","last_change":"2024-03-11T21:18:52.263997+0000","last_active":"2024-03-11T21:19:18.677518+0000","last_peered":"2024-03-11T21:19:18.677518+0000","last_clean":"2024-03-11T21:19:18.677518+0000","last_became_active":"2024-03-11T21:18:52.263016+0000","last_became_peered":"2024-03-11T21:18:52.263016+0000","last_unstale":"2024-03-11T21:19:18.677518+0000","last_undegraded":"2024-03-11T21:19:18.677518+0000","last_fullsized":"2024-03-11T21:19:18.677518+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:48.586572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673322+0000","last_change":"2024-03-11T21:18:48.244043+0000","last_active":"2024-03-11T21:19:18.673322+0000","last_peered":"2024-03-11T21:19:18.673322+0000","last_clean":"2024-03-11T21:19:18.673322+0000","last_became_active":"2024-03-11T21:18:48.243558+0000","last_became_peered":"2024-03-11T21:18:48.243558+0000","last_unstale":"2024-03-11T21:19:18.673322+0000","last_undegraded":"2024-03-11T21:19:18.673322+0000","last_fullsized":"2024-03-11T21:19:18.673322+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:04.521470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673777+0000","last_change":"2024-03-11T21:18:50.254872+0000","last_active":"2024-03-11T21:19:18.673777+0000","last_peered":"2024-03-11T21:19:18.673777+0000","last_clean":"2024-03-11T21:19:18.673777+0000","last_became_active":"2024-03-11T21:18:50.254569+0000","last_became_peered":"2024-03-11T21:18:50.254569+0000","last_unstale":"2024-03-11T21:19:18.673777+0000","last_undegraded":"2024-03-11T21:19:18.673777+0000","last_fullsized":"2024-03-11T21:19:18.673777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:05:47.625061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675049+0000","last_change":"2024-03-11T21:18:54.270024+0000","last_active":"2024-03-11T21:19:18.675049+0000","last_peered":"2024-03-11T21:19:18.675049+0000","last_clean":"2024-03-11T21:19:18.675049+0000","last_became_active":"2024-03-11T21:18:54.269724+0000","last_became_peered":"2024-03-11T21:18:54.269724+0000","last_unstale":"2024-03-11T21:19:18.675049+0000","last_undegraded":"2024-03-11T21:19:18.675049+0000","last_fullsized":"2024-03-11T21:19:18.675049+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:58.156964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674559+0000","last_change":"2024-03-11T21:18:48.247181+0000","last_active":"2024-03-11T21:19:18.674559+0000","last_peered":"2024-03-11T21:19:18.674559+0000","last_clean":"2024-03-11T21:19:18.674559+0000","last_became_active":"2024-03-11T21:18:48.246859+0000","last_became_peered":"2024-03-11T21:18:48.246859+0000","last_unstale":"2024-03-11T21:19:18.674559+0000","last_undegraded":"2024-03-11T21:19:18.674559+0000","last_fullsized":"2024-03-11T21:19:18.674559+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:53.677395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673374+0000","last_change":"2024-03-11T21:18:50.253471+0000","last_active":"2024-03-11T21:19:18.673374+0000","last_peered":"2024-03-11T21:19:18.673374+0000","last_clean":"2024-03-11T21:19:18.673374+0000","last_became_active":"2024-03-11T21:18:50.252846+0000","last_became_peered":"2024-03-11T21:18:50.252846+0000","last_unstale":"2024-03-11T21:19:18.673374+0000","last_undegraded":"2024-03-11T21:19:18.673374+0000","last_fullsized":"2024-03-11T21:19:18.673374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:25:40.921942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674433+0000","last_change":"2024-03-11T21:18:52.259569+0000","last_active":"2024-03-11T21:19:18.674433+0000","last_peered":"2024-03-11T21:19:18.674433+0000","last_clean":"2024-03-11T21:19:18.674433+0000","last_became_active":"2024-03-11T21:18:52.259095+0000","last_became_peered":"2024-03-11T21:18:52.259095+0000","last_unstale":"2024-03-11T21:19:18.674433+0000","last_undegraded":"2024-03-11T21:19:18.674433+0000","last_fullsized":"2024-03-11T21:19:18.674433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:01.981684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674986+0000","last_change":"2024-03-11T21:18:54.270644+0000","last_active":"2024-03-11T21:19:18.674986+0000","last_peered":"2024-03-11T21:19:18.674986+0000","last_clean":"2024-03-11T21:19:18.674986+0000","last_became_active":"2024-03-11T21:18:54.270107+0000","last_became_peered":"2024-03-11T21:18:54.270107+0000","last_unstale":"2024-03-11T21:19:18.674986+0000","last_undegraded":"2024-03-11T21:19:18.674986+0000","last_fullsized":"2024-03-11T21:19:18.674986+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:13.964645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":36,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:23.295671+0000","last_change":"2024-03-11T21:18:52.261525+0000","last_active":"2024-03-11T21:19:23.295671+0000","last_peered":"2024-03-11T21:19:23.295671+0000","last_clean":"2024-03-11T21:19:23.295671+0000","last_became_active":"2024-03-11T21:18:52.261203+0000","last_became_peered":"2024-03-11T21:18:52.261203+0000","last_unstale":"2024-03-11T21:19:23.295671+0000","last_undegraded":"2024-03-11T21:19:23.295671+0000","last_fullsized":"2024-03-11T21:19:23.295671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:59:52.473583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674745+0000","last_change":"2024-03-11T21:18:48.250700+0000","last_active":"2024-03-11T21:19:18.674745+0000","last_peered":"2024-03-11T21:19:18.674745+0000","last_clean":"2024-03-11T21:19:18.674745+0000","last_became_active":"2024-03-11T21:18:48.250248+0000","last_became_peered":"2024-03-11T21:18:48.250248+0000","last_unstale":"2024-03-11T21:19:18.674745+0000","last_undegraded":"2024-03-11T21:19:18.674745+0000","last_fullsized":"2024-03-11T21:19:18.674745+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:45.525049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675242+0000","last_change":"2024-03-11T21:18:50.253308+0000","last_active":"2024-03-11T21:19:18.675242+0000","last_peered":"2024-03-11T21:19:18.675242+0000","last_clean":"2024-03-11T21:19:18.675242+0000","last_became_active":"2024-03-11T21:18:50.251849+0000","last_became_peered":"2024-03-11T21:18:50.251849+0000","last_unstale":"2024-03-11T21:19:18.675242+0000","last_undegraded":"2024-03-11T21:19:18.675242+0000","last_fullsized":"2024-03-11T21:19:18.675242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:44:03.848772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674527+0000","last_change":"2024-03-11T21:18:54.272927+0000","last_active":"2024-03-11T21:19:18.674527+0000","last_peered":"2024-03-11T21:19:18.674527+0000","last_clean":"2024-03-11T21:19:18.674527+0000","last_became_active":"2024-03-11T21:18:54.272552+0000","last_became_peered":"2024-03-11T21:18:54.272552+0000","last_unstale":"2024-03-11T21:19:18.674527+0000","last_undegraded":"2024-03-11T21:19:18.674527+0000","last_fullsized":"2024-03-11T21:19:18.674527+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:33:47.842278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676257+0000","last_change":"2024-03-11T21:18:48.247121+0000","last_active":"2024-03-11T21:19:18.676257+0000","last_peered":"2024-03-11T21:19:18.676257+0000","last_clean":"2024-03-11T21:19:18.676257+0000","last_became_active":"2024-03-11T21:18:48.246827+0000","last_became_peered":"2024-03-11T21:18:48.246827+0000","last_unstale":"2024-03-11T21:19:18.676257+0000","last_undegraded":"2024-03-11T21:19:18.676257+0000","last_fullsized":"2024-03-11T21:19:18.676257+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:06.460925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676114+0000","last_change":"2024-03-11T21:18:50.255062+0000","last_active":"2024-03-11T21:19:18.676114+0000","last_peered":"2024-03-11T21:19:18.676114+0000","last_clean":"2024-03-11T21:19:18.676114+0000","last_became_active":"2024-03-11T21:18:50.253579+0000","last_became_peered":"2024-03-11T21:18:50.253579+0000","last_unstale":"2024-03-11T21:19:18.676114+0000","last_undegraded":"2024-03-11T21:19:18.676114+0000","last_fullsized":"2024-03-11T21:19:18.676114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:23.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673722+0000","last_change":"2024-03-11T21:18:52.257009+0000","last_active":"2024-03-11T21:19:18.673722+0000","last_peered":"2024-03-11T21:19:18.673722+0000","last_clean":"2024-03-11T21:19:18.673722+0000","last_became_active":"2024-03-11T21:18:52.256109+0000","last_became_peered":"2024-03-11T21:18:52.256109+0000","last_unstale":"2024-03-11T21:19:18.673722+0000","last_undegraded":"2024-03-11T21:19:18.673722+0000","last_fullsized":"2024-03-11T21:19:18.673722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:49:52.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673580+0000","last_change":"2024-03-11T21:18:54.268744+0000","last_active":"2024-03-11T21:19:18.673580+0000","last_peered":"2024-03-11T21:19:18.673580+0000","last_clean":"2024-03-11T21:19:18.673580+0000","last_became_active":"2024-03-11T21:18:54.267967+0000","last_became_peered":"2024-03-11T21:18:54.267967+0000","last_unstale":"2024-03-11T21:19:18.673580+0000","last_undegraded":"2024-03-11T21:19:18.673580+0000","last_fullsized":"2024-03-11T21:19:18.673580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:24:23.737447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675364+0000","last_change":"2024-03-11T21:18:48.250517+0000","last_active":"2024-03-11T21:19:18.675364+0000","last_peered":"2024-03-11T21:19:18.675364+0000","last_clean":"2024-03-11T21:19:18.675364+0000","last_became_active":"2024-03-11T21:18:48.250220+0000","last_became_peered":"2024-03-11T21:18:48.250220+0000","last_unstale":"2024-03-11T21:19:18.675364+0000","last_undegraded":"2024-03-11T21:19:18.675364+0000","last_fullsized":"2024-03-11T21:19:18.675364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:25:15.099637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675267+0000","last_change":"2024-03-11T21:18:50.256216+0000","last_active":"2024-03-11T21:19:18.675267+0000","last_peered":"2024-03-11T21:19:18.675267+0000","last_clean":"2024-03-11T21:19:18.675267+0000","last_became_active":"2024-03-11T21:18:50.255795+0000","last_became_peered":"2024-03-11T21:18:50.255795+0000","last_unstale":"2024-03-11T21:19:18.675267+0000","last_undegraded":"2024-03-11T21:19:18.675267+0000","last_fullsized":"2024-03-11T21:19:18.675267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:08.862808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673122+0000","last_change":"2024-03-11T21:18:52.259482+0000","last_active":"2024-03-11T21:19:18.673122+0000","last_peered":"2024-03-11T21:19:18.673122+0000","last_clean":"2024-03-11T21:19:18.673122+0000","last_became_active":"2024-03-11T21:18:52.258897+0000","last_became_peered":"2024-03-11T21:18:52.258897+0000","last_unstale":"2024-03-11T21:19:18.673122+0000","last_undegraded":"2024-03-11T21:19:18.673122+0000","last_fullsized":"2024-03-11T21:19:18.673122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:26.180192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673455+0000","last_change":"2024-03-11T21:18:54.270670+0000","last_active":"2024-03-11T21:19:18.673455+0000","last_peered":"2024-03-11T21:19:18.673455+0000","last_clean":"2024-03-11T21:19:18.673455+0000","last_became_active":"2024-03-11T21:18:54.269378+0000","last_became_peered":"2024-03-11T21:18:54.269378+0000","last_unstale":"2024-03-11T21:19:18.673455+0000","last_undegraded":"2024-03-11T21:19:18.673455+0000","last_fullsized":"2024-03-11T21:19:18.673455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:24:09.119334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676842+0000","last_change":"2024-03-11T21:18:48.245458+0000","last_active":"2024-03-11T21:19:18.676842+0000","last_peered":"2024-03-11T21:19:18.676842+0000","last_clean":"2024-03-11T21:19:18.676842+0000","last_became_active":"2024-03-11T21:18:48.243569+0000","last_became_peered":"2024-03-11T21:18:48.243569+0000","last_unstale":"2024-03-11T21:19:18.676842+0000","last_undegraded":"2024-03-11T21:19:18.676842+0000","last_fullsized":"2024-03-11T21:19:18.676842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:57.823070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675859+0000","last_change":"2024-03-11T21:18:50.253928+0000","last_active":"2024-03-11T21:19:18.675859+0000","last_peered":"2024-03-11T21:19:18.675859+0000","last_clean":"2024-03-11T21:19:18.675859+0000","last_became_active":"2024-03-11T21:18:50.253429+0000","last_became_peered":"2024-03-11T21:18:50.253429+0000","last_unstale":"2024-03-11T21:19:18.675859+0000","last_undegraded":"2024-03-11T21:19:18.675859+0000","last_fullsized":"2024-03-11T21:19:18.675859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:56:27.094231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673869+0000","last_change":"2024-03-11T21:18:52.261308+0000","last_active":"2024-03-11T21:19:18.673869+0000","last_peered":"2024-03-11T21:19:18.673869+0000","last_clean":"2024-03-11T21:19:18.673869+0000","last_became_active":"2024-03-11T21:18:52.260499+0000","last_became_peered":"2024-03-11T21:18:52.260499+0000","last_unstale":"2024-03-11T21:19:18.673869+0000","last_undegraded":"2024-03-11T21:19:18.673869+0000","last_fullsized":"2024-03-11T21:19:18.673869+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:32:59.436003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:19:31.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:19:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:32.590 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:32 smithi001 ceph-mon[25774]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:19:32.590 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:32 smithi001 ceph-mon[25774]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:19:32.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:32 smithi017 ceph-mon[29862]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:19:32.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:32 smithi017 ceph-mon[29862]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:19:32.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:32 smithi204 ceph-mon[30291]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:19:32.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:32 smithi204 ceph-mon[30291]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:19:34.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:34 smithi001 ceph-mon[25774]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:19:34.456 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:34 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:34 smithi017 ceph-mon[29862]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:19:34.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:34 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:34.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:34 smithi204 ceph-mon[30291]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:19:34.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:34 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:35.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:19:35.776 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-03-11T21:19:36.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:36 smithi017 ceph-mon[29862]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:19:36.705 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:36 smithi001 ceph-mon[25774]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:19:36.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:36 smithi204 ceph-mon[30291]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:19:37.491 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-11T21:19:34.687969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595984,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":95743312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674270+0000","last_change":"2024-03-11T21:18:54.268801+0000","last_active":"2024-03-11T21:19:18.674270+0000","last_peered":"2024-03-11T21:19:18.674270+0000","last_clean":"2024-03-11T21:19:18.674270+0000","last_became_active":"2024-03-11T21:18:54.267796+0000","last_became_peered":"2024-03-11T21:18:54.267796+0000","last_unstale":"2024-03-11T21:19:18.674270+0000","last_undegraded":"2024-03-11T21:19:18.674270+0000","last_fullsized":"2024-03-11T21:19:18.674270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:25:59.917624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673557+0000","last_change":"2024-03-11T21:18:52.259801+0000","last_active":"2024-03-11T21:19:18.673557+0000","last_peered":"2024-03-11T21:19:18.673557+0000","last_clean":"2024-03-11T21:19:18.673557+0000","last_became_active":"2024-03-11T21:18:52.259449+0000","last_became_peered":"2024-03-11T21:18:52.259449+0000","last_unstale":"2024-03-11T21:19:18.673557+0000","last_undegraded":"2024-03-11T21:19:18.673557+0000","last_fullsized":"2024-03-11T21:19:18.673557+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:18:18.356017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677943+0000","last_change":"2024-03-11T21:18:48.244934+0000","last_active":"2024-03-11T21:19:18.677943+0000","last_peered":"2024-03-11T21:19:18.677943+0000","last_clean":"2024-03-11T21:19:18.677943+0000","last_became_active":"2024-03-11T21:18:48.243621+0000","last_became_peered":"2024-03-11T21:18:48.243621+0000","last_unstale":"2024-03-11T21:19:18.677943+0000","last_undegraded":"2024-03-11T21:19:18.677943+0000","last_fullsized":"2024-03-11T21:19:18.677943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:21:41.655490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676701+0000","last_change":"2024-03-11T21:18:50.258500+0000","last_active":"2024-03-11T21:19:18.676701+0000","last_peered":"2024-03-11T21:19:18.676701+0000","last_clean":"2024-03-11T21:19:18.676701+0000","last_became_active":"2024-03-11T21:18:50.257355+0000","last_became_peered":"2024-03-11T21:18:50.257355+0000","last_unstale":"2024-03-11T21:19:18.676701+0000","last_undegraded":"2024-03-11T21:19:18.676701+0000","last_fullsized":"2024-03-11T21:19:18.676701+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:43:14.949706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673225+0000","last_change":"2024-03-11T21:18:54.270030+0000","last_active":"2024-03-11T21:19:18.673225+0000","last_peered":"2024-03-11T21:19:18.673225+0000","last_clean":"2024-03-11T21:19:18.673225+0000","last_became_active":"2024-03-11T21:18:54.269284+0000","last_became_peered":"2024-03-11T21:18:54.269284+0000","last_unstale":"2024-03-11T21:19:18.673225+0000","last_undegraded":"2024-03-11T21:19:18.673225+0000","last_fullsized":"2024-03-11T21:19:18.673225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:33.322697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673782+0000","last_change":"2024-03-11T21:18:52.258854+0000","last_active":"2024-03-11T21:19:18.673782+0000","last_peered":"2024-03-11T21:19:18.673782+0000","last_clean":"2024-03-11T21:19:18.673782+0000","last_became_active":"2024-03-11T21:18:52.257924+0000","last_became_peered":"2024-03-11T21:18:52.257924+0000","last_unstale":"2024-03-11T21:19:18.673782+0000","last_undegraded":"2024-03-11T21:19:18.673782+0000","last_fullsized":"2024-03-11T21:19:18.673782+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:15.300210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677025+0000","last_change":"2024-03-11T21:18:48.245070+0000","last_active":"2024-03-11T21:19:18.677025+0000","last_peered":"2024-03-11T21:19:18.677025+0000","last_clean":"2024-03-11T21:19:18.677025+0000","last_became_active":"2024-03-11T21:18:48.243854+0000","last_became_peered":"2024-03-11T21:18:48.243854+0000","last_unstale":"2024-03-11T21:19:18.677025+0000","last_undegraded":"2024-03-11T21:19:18.677025+0000","last_fullsized":"2024-03-11T21:19:18.677025+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:22.577485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677555+0000","last_change":"2024-03-11T21:18:50.256291+0000","last_active":"2024-03-11T21:19:18.677555+0000","last_peered":"2024-03-11T21:19:18.677555+0000","last_clean":"2024-03-11T21:19:18.677555+0000","last_became_active":"2024-03-11T21:18:50.255871+0000","last_became_peered":"2024-03-11T21:18:50.255871+0000","last_unstale":"2024-03-11T21:19:18.677555+0000","last_undegraded":"2024-03-11T21:19:18.677555+0000","last_fullsized":"2024-03-11T21:19:18.677555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:15:10.582718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673760+0000","last_change":"2024-03-11T21:18:54.269077+0000","last_active":"2024-03-11T21:19:18.673760+0000","last_peered":"2024-03-11T21:19:18.673760+0000","last_clean":"2024-03-11T21:19:18.673760+0000","last_became_active":"2024-03-11T21:18:54.268687+0000","last_became_peered":"2024-03-11T21:18:54.268687+0000","last_unstale":"2024-03-11T21:19:18.673760+0000","last_undegraded":"2024-03-11T21:19:18.673760+0000","last_fullsized":"2024-03-11T21:19:18.673760+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:29.930763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674184+0000","last_change":"2024-03-11T21:18:52.257886+0000","last_active":"2024-03-11T21:19:18.674184+0000","last_peered":"2024-03-11T21:19:18.674184+0000","last_clean":"2024-03-11T21:19:18.674184+0000","last_became_active":"2024-03-11T21:18:52.257495+0000","last_became_peered":"2024-03-11T21:18:52.257495+0000","last_unstale":"2024-03-11T21:19:18.674184+0000","last_undegraded":"2024-03-11T21:19:18.674184+0000","last_fullsized":"2024-03-11T21:19:18.674184+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:43.742084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675804+0000","last_change":"2024-03-11T21:18:48.247988+0000","last_active":"2024-03-11T21:19:18.675804+0000","last_peered":"2024-03-11T21:19:18.675804+0000","last_clean":"2024-03-11T21:19:18.675804+0000","last_became_active":"2024-03-11T21:18:48.247716+0000","last_became_peered":"2024-03-11T21:18:48.247716+0000","last_unstale":"2024-03-11T21:19:18.675804+0000","last_undegraded":"2024-03-11T21:19:18.675804+0000","last_fullsized":"2024-03-11T21:19:18.675804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:52:43.612188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675321+0000","last_change":"2024-03-11T21:18:50.256296+0000","last_active":"2024-03-11T21:19:18.675321+0000","last_peered":"2024-03-11T21:19:18.675321+0000","last_clean":"2024-03-11T21:19:18.675321+0000","last_became_active":"2024-03-11T21:18:50.255951+0000","last_became_peered":"2024-03-11T21:18:50.255951+0000","last_unstale":"2024-03-11T21:19:18.675321+0000","last_undegraded":"2024-03-11T21:19:18.675321+0000","last_fullsized":"2024-03-11T21:19:18.675321+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:03.095698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674156+0000","last_change":"2024-03-11T21:18:54.270704+0000","last_active":"2024-03-11T21:19:18.674156+0000","last_peered":"2024-03-11T21:19:18.674156+0000","last_clean":"2024-03-11T21:19:18.674156+0000","last_became_active":"2024-03-11T21:18:54.269555+0000","last_became_peered":"2024-03-11T21:18:54.269555+0000","last_unstale":"2024-03-11T21:19:18.674156+0000","last_undegraded":"2024-03-11T21:19:18.674156+0000","last_fullsized":"2024-03-11T21:19:18.674156+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:51.587929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673536+0000","last_change":"2024-03-11T21:18:52.257020+0000","last_active":"2024-03-11T21:19:18.673536+0000","last_peered":"2024-03-11T21:19:18.673536+0000","last_clean":"2024-03-11T21:19:18.673536+0000","last_became_active":"2024-03-11T21:18:52.256248+0000","last_became_peered":"2024-03-11T21:18:52.256248+0000","last_unstale":"2024-03-11T21:19:18.673536+0000","last_undegraded":"2024-03-11T21:19:18.673536+0000","last_fullsized":"2024-03-11T21:19:18.673536+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:53.234541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676123+0000","last_change":"2024-03-11T21:18:48.247232+0000","last_active":"2024-03-11T21:19:18.676123+0000","last_peered":"2024-03-11T21:19:18.676123+0000","last_clean":"2024-03-11T21:19:18.676123+0000","last_became_active":"2024-03-11T21:18:48.246964+0000","last_became_peered":"2024-03-11T21:18:48.246964+0000","last_unstale":"2024-03-11T21:19:18.676123+0000","last_undegraded":"2024-03-11T21:19:18.676123+0000","last_fullsized":"2024-03-11T21:19:18.676123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:19.772442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675662+0000","last_change":"2024-03-11T21:18:50.255375+0000","last_active":"2024-03-11T21:19:18.675662+0000","last_peered":"2024-03-11T21:19:18.675662+0000","last_clean":"2024-03-11T21:19:18.675662+0000","last_became_active":"2024-03-11T21:18:50.255028+0000","last_became_peered":"2024-03-11T21:18:50.255028+0000","last_unstale":"2024-03-11T21:19:18.675662+0000","last_undegraded":"2024-03-11T21:19:18.675662+0000","last_fullsized":"2024-03-11T21:19:18.675662+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:51:54.280374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'2","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674307+0000","last_change":"2024-03-11T21:18:54.270647+0000","last_active":"2024-03-11T21:19:18.674307+0000","last_peered":"2024-03-11T21:19:18.674307+0000","last_clean":"2024-03-11T21:19:18.674307+0000","last_became_active":"2024-03-11T21:18:54.270206+0000","last_became_peered":"2024-03-11T21:18:54.270206+0000","last_unstale":"2024-03-11T21:19:18.674307+0000","last_undegraded":"2024-03-11T21:19:18.674307+0000","last_fullsized":"2024-03-11T21:19:18.674307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:23.514703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675133+0000","last_change":"2024-03-11T21:18:52.262335+0000","last_active":"2024-03-11T21:19:18.675133+0000","last_peered":"2024-03-11T21:19:18.675133+0000","last_clean":"2024-03-11T21:19:18.675133+0000","last_became_active":"2024-03-11T21:18:52.261325+0000","last_became_peered":"2024-03-11T21:18:52.261325+0000","last_unstale":"2024-03-11T21:19:18.675133+0000","last_undegraded":"2024-03-11T21:19:18.675133+0000","last_fullsized":"2024-03-11T21:19:18.675133+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:13:26.819492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675245+0000","last_change":"2024-03-11T21:18:48.247116+0000","last_active":"2024-03-11T21:19:18.675245+0000","last_peered":"2024-03-11T21:19:18.675245+0000","last_clean":"2024-03-11T21:19:18.675245+0000","last_became_active":"2024-03-11T21:18:48.246781+0000","last_became_peered":"2024-03-11T21:18:48.246781+0000","last_unstale":"2024-03-11T21:19:18.675245+0000","last_undegraded":"2024-03-11T21:19:18.675245+0000","last_fullsized":"2024-03-11T21:19:18.675245+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:05.229493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674678+0000","last_change":"2024-03-11T21:18:50.255114+0000","last_active":"2024-03-11T21:19:18.674678+0000","last_peered":"2024-03-11T21:19:18.674678+0000","last_clean":"2024-03-11T21:19:18.674678+0000","last_became_active":"2024-03-11T21:18:50.254811+0000","last_became_peered":"2024-03-11T21:18:50.254811+0000","last_unstale":"2024-03-11T21:19:18.674678+0000","last_undegraded":"2024-03-11T21:19:18.674678+0000","last_fullsized":"2024-03-11T21:19:18.674678+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:46.755255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675788+0000","last_change":"2024-03-11T21:18:54.270901+0000","last_active":"2024-03-11T21:19:18.675788+0000","last_peered":"2024-03-11T21:19:18.675788+0000","last_clean":"2024-03-11T21:19:18.675788+0000","last_became_active":"2024-03-11T21:18:54.270175+0000","last_became_peered":"2024-03-11T21:18:54.270175+0000","last_unstale":"2024-03-11T21:19:18.675788+0000","last_undegraded":"2024-03-11T21:19:18.675788+0000","last_fullsized":"2024-03-11T21:19:18.675788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:35:19.940205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676018+0000","last_change":"2024-03-11T21:18:52.261762+0000","last_active":"2024-03-11T21:19:18.676018+0000","last_peered":"2024-03-11T21:19:18.676018+0000","last_clean":"2024-03-11T21:19:18.676018+0000","last_became_active":"2024-03-11T21:18:52.261101+0000","last_became_peered":"2024-03-11T21:18:52.261101+0000","last_unstale":"2024-03-11T21:19:18.676018+0000","last_undegraded":"2024-03-11T21:19:18.676018+0000","last_fullsized":"2024-03-11T21:19:18.676018+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:48.132420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674179+0000","last_change":"2024-03-11T21:18:48.247627+0000","last_active":"2024-03-11T21:19:18.674179+0000","last_peered":"2024-03-11T21:19:18.674179+0000","last_clean":"2024-03-11T21:19:18.674179+0000","last_became_active":"2024-03-11T21:18:48.247098+0000","last_became_peered":"2024-03-11T21:18:48.247098+0000","last_unstale":"2024-03-11T21:19:18.674179+0000","last_undegraded":"2024-03-11T21:19:18.674179+0000","last_fullsized":"2024-03-11T21:19:18.674179+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:32:42.106334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675079+0000","last_change":"2024-03-11T21:18:50.256293+0000","last_active":"2024-03-11T21:19:18.675079+0000","last_peered":"2024-03-11T21:19:18.675079+0000","last_clean":"2024-03-11T21:19:18.675079+0000","last_became_active":"2024-03-11T21:18:50.255976+0000","last_became_peered":"2024-03-11T21:18:50.255976+0000","last_unstale":"2024-03-11T21:19:18.675079+0000","last_undegraded":"2024-03-11T21:19:18.675079+0000","last_fullsized":"2024-03-11T21:19:18.675079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:44:32.787829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676180+0000","last_change":"2024-03-11T21:18:54.271422+0000","last_active":"2024-03-11T21:19:18.676180+0000","last_peered":"2024-03-11T21:19:18.676180+0000","last_clean":"2024-03-11T21:19:18.676180+0000","last_became_active":"2024-03-11T21:18:54.271108+0000","last_became_peered":"2024-03-11T21:18:54.271108+0000","last_unstale":"2024-03-11T21:19:18.676180+0000","last_undegraded":"2024-03-11T21:19:18.676180+0000","last_fullsized":"2024-03-11T21:19:18.676180+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:45:41.196419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296646+0000","last_change":"2024-03-11T21:18:52.262431+0000","last_active":"2024-03-11T21:19:28.296646+0000","last_peered":"2024-03-11T21:19:28.296646+0000","last_clean":"2024-03-11T21:19:28.296646+0000","last_became_active":"2024-03-11T21:18:52.262116+0000","last_became_peered":"2024-03-11T21:18:52.262116+0000","last_unstale":"2024-03-11T21:19:28.296646+0000","last_undegraded":"2024-03-11T21:19:28.296646+0000","last_fullsized":"2024-03-11T21:19:28.296646+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:22:34.117731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673445+0000","last_change":"2024-03-11T21:18:48.244992+0000","last_active":"2024-03-11T21:19:18.673445+0000","last_peered":"2024-03-11T21:19:18.673445+0000","last_clean":"2024-03-11T21:19:18.673445+0000","last_became_active":"2024-03-11T21:18:48.243766+0000","last_became_peered":"2024-03-11T21:18:48.243766+0000","last_unstale":"2024-03-11T21:19:18.673445+0000","last_undegraded":"2024-03-11T21:19:18.673445+0000","last_fullsized":"2024-03-11T21:19:18.673445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:45:09.893389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672551+0000","last_change":"2024-03-11T21:18:50.256193+0000","last_active":"2024-03-11T21:19:18.672551+0000","last_peered":"2024-03-11T21:19:18.672551+0000","last_clean":"2024-03-11T21:19:18.672551+0000","last_became_active":"2024-03-11T21:18:50.253709+0000","last_became_peered":"2024-03-11T21:18:50.253709+0000","last_unstale":"2024-03-11T21:19:18.672551+0000","last_undegraded":"2024-03-11T21:19:18.672551+0000","last_fullsized":"2024-03-11T21:19:18.672551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:25.617190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674638+0000","last_change":"2024-03-11T21:18:54.272403+0000","last_active":"2024-03-11T21:19:18.674638+0000","last_peered":"2024-03-11T21:19:18.674638+0000","last_clean":"2024-03-11T21:19:18.674638+0000","last_became_active":"2024-03-11T21:18:54.271448+0000","last_became_peered":"2024-03-11T21:18:54.271448+0000","last_unstale":"2024-03-11T21:19:18.674638+0000","last_undegraded":"2024-03-11T21:19:18.674638+0000","last_fullsized":"2024-03-11T21:19:18.674638+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:30:03.377329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676347+0000","last_change":"2024-03-11T21:18:52.263648+0000","last_active":"2024-03-11T21:19:18.676347+0000","last_peered":"2024-03-11T21:19:18.676347+0000","last_clean":"2024-03-11T21:19:18.676347+0000","last_became_active":"2024-03-11T21:18:52.263265+0000","last_became_peered":"2024-03-11T21:18:52.263265+0000","last_unstale":"2024-03-11T21:19:18.676347+0000","last_undegraded":"2024-03-11T21:19:18.676347+0000","last_fullsized":"2024-03-11T21:19:18.676347+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:36:35.868987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673636+0000","last_change":"2024-03-11T21:18:48.248405+0000","last_active":"2024-03-11T21:19:18.673636+0000","last_peered":"2024-03-11T21:19:18.673636+0000","last_clean":"2024-03-11T21:19:18.673636+0000","last_became_active":"2024-03-11T21:18:48.248131+0000","last_became_peered":"2024-03-11T21:18:48.248131+0000","last_unstale":"2024-03-11T21:19:18.673636+0000","last_undegraded":"2024-03-11T21:19:18.673636+0000","last_fullsized":"2024-03-11T21:19:18.673636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:23:54.755461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672890+0000","last_change":"2024-03-11T21:18:50.253219+0000","last_active":"2024-03-11T21:19:18.672890+0000","last_peered":"2024-03-11T21:19:18.672890+0000","last_clean":"2024-03-11T21:19:18.672890+0000","last_became_active":"2024-03-11T21:18:50.252031+0000","last_became_peered":"2024-03-11T21:18:50.252031+0000","last_unstale":"2024-03-11T21:19:18.672890+0000","last_undegraded":"2024-03-11T21:19:18.672890+0000","last_fullsized":"2024-03-11T21:19:18.672890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:48:04.913293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674460+0000","last_change":"2024-03-11T21:18:54.271299+0000","last_active":"2024-03-11T21:19:18.674460+0000","last_peered":"2024-03-11T21:19:18.674460+0000","last_clean":"2024-03-11T21:19:18.674460+0000","last_became_active":"2024-03-11T21:18:54.270919+0000","last_became_peered":"2024-03-11T21:18:54.270919+0000","last_unstale":"2024-03-11T21:19:18.674460+0000","last_undegraded":"2024-03-11T21:19:18.674460+0000","last_fullsized":"2024-03-11T21:19:18.674460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:54:46.145276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672713+0000","last_change":"2024-03-11T21:18:52.258529+0000","last_active":"2024-03-11T21:19:18.672713+0000","last_peered":"2024-03-11T21:19:18.672713+0000","last_clean":"2024-03-11T21:19:18.672713+0000","last_became_active":"2024-03-11T21:18:52.257868+0000","last_became_peered":"2024-03-11T21:18:52.257868+0000","last_unstale":"2024-03-11T21:19:18.672713+0000","last_undegraded":"2024-03-11T21:19:18.672713+0000","last_fullsized":"2024-03-11T21:19:18.672713+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:04.556993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676091+0000","last_change":"2024-03-11T21:18:48.248882+0000","last_active":"2024-03-11T21:19:18.676091+0000","last_peered":"2024-03-11T21:19:18.676091+0000","last_clean":"2024-03-11T21:19:18.676091+0000","last_became_active":"2024-03-11T21:18:48.248283+0000","last_became_peered":"2024-03-11T21:18:48.248283+0000","last_unstale":"2024-03-11T21:19:18.676091+0000","last_undegraded":"2024-03-11T21:19:18.676091+0000","last_fullsized":"2024-03-11T21:19:18.676091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:16.551148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675950+0000","last_change":"2024-03-11T21:18:50.254073+0000","last_active":"2024-03-11T21:19:18.675950+0000","last_peered":"2024-03-11T21:19:18.675950+0000","last_clean":"2024-03-11T21:19:18.675950+0000","last_became_active":"2024-03-11T21:18:50.253420+0000","last_became_peered":"2024-03-11T21:18:50.253420+0000","last_unstale":"2024-03-11T21:19:18.675950+0000","last_undegraded":"2024-03-11T21:19:18.675950+0000","last_fullsized":"2024-03-11T21:19:18.675950+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:51.329348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673920+0000","last_change":"2024-03-11T21:18:54.270785+0000","last_active":"2024-03-11T21:19:18.673920+0000","last_peered":"2024-03-11T21:19:18.673920+0000","last_clean":"2024-03-11T21:19:18.673920+0000","last_became_active":"2024-03-11T21:18:54.270118+0000","last_became_peered":"2024-03-11T21:18:54.270118+0000","last_unstale":"2024-03-11T21:19:18.673920+0000","last_undegraded":"2024-03-11T21:19:18.673920+0000","last_fullsized":"2024-03-11T21:19:18.673920+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:48.827359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672889+0000","last_change":"2024-03-11T21:18:52.258813+0000","last_active":"2024-03-11T21:19:18.672889+0000","last_peered":"2024-03-11T21:19:18.672889+0000","last_clean":"2024-03-11T21:19:18.672889+0000","last_became_active":"2024-03-11T21:18:52.258463+0000","last_became_peered":"2024-03-11T21:18:52.258463+0000","last_unstale":"2024-03-11T21:19:18.672889+0000","last_undegraded":"2024-03-11T21:19:18.672889+0000","last_fullsized":"2024-03-11T21:19:18.672889+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:23:11.275587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675738+0000","last_change":"2024-03-11T21:18:48.248497+0000","last_active":"2024-03-11T21:19:18.675738+0000","last_peered":"2024-03-11T21:19:18.675738+0000","last_clean":"2024-03-11T21:19:18.675738+0000","last_became_active":"2024-03-11T21:18:48.248121+0000","last_became_peered":"2024-03-11T21:18:48.248121+0000","last_unstale":"2024-03-11T21:19:18.675738+0000","last_undegraded":"2024-03-11T21:19:18.675738+0000","last_fullsized":"2024-03-11T21:19:18.675738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:01.527285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677731+0000","last_change":"2024-03-11T21:18:50.256569+0000","last_active":"2024-03-11T21:19:18.677731+0000","last_peered":"2024-03-11T21:19:18.677731+0000","last_clean":"2024-03-11T21:19:18.677731+0000","last_became_active":"2024-03-11T21:18:50.256060+0000","last_became_peered":"2024-03-11T21:18:50.256060+0000","last_unstale":"2024-03-11T21:19:18.677731+0000","last_undegraded":"2024-03-11T21:19:18.677731+0000","last_fullsized":"2024-03-11T21:19:18.677731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:49:08.755492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674773+0000","last_change":"2024-03-11T21:18:54.268729+0000","last_active":"2024-03-11T21:19:18.674773+0000","last_peered":"2024-03-11T21:19:18.674773+0000","last_clean":"2024-03-11T21:19:18.674773+0000","last_became_active":"2024-03-11T21:18:54.268431+0000","last_became_peered":"2024-03-11T21:18:54.268431+0000","last_unstale":"2024-03-11T21:19:18.674773+0000","last_undegraded":"2024-03-11T21:19:18.674773+0000","last_fullsized":"2024-03-11T21:19:18.674773+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:42:23.684054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674300+0000","last_change":"2024-03-11T21:18:52.262140+0000","last_active":"2024-03-11T21:19:18.674300+0000","last_peered":"2024-03-11T21:19:18.674300+0000","last_clean":"2024-03-11T21:19:18.674300+0000","last_became_active":"2024-03-11T21:18:52.261854+0000","last_became_peered":"2024-03-11T21:18:52.261854+0000","last_unstale":"2024-03-11T21:19:18.674300+0000","last_undegraded":"2024-03-11T21:19:18.674300+0000","last_fullsized":"2024-03-11T21:19:18.674300+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:13.199921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675658+0000","last_change":"2024-03-11T21:18:48.250770+0000","last_active":"2024-03-11T21:19:18.675658+0000","last_peered":"2024-03-11T21:19:18.675658+0000","last_clean":"2024-03-11T21:19:18.675658+0000","last_became_active":"2024-03-11T21:18:48.250240+0000","last_became_peered":"2024-03-11T21:18:48.250240+0000","last_unstale":"2024-03-11T21:19:18.675658+0000","last_undegraded":"2024-03-11T21:19:18.675658+0000","last_fullsized":"2024-03-11T21:19:18.675658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:27:40.316142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676705+0000","last_change":"2024-03-11T21:18:50.254249+0000","last_active":"2024-03-11T21:19:18.676705+0000","last_peered":"2024-03-11T21:19:18.676705+0000","last_clean":"2024-03-11T21:19:18.676705+0000","last_became_active":"2024-03-11T21:18:50.253381+0000","last_became_peered":"2024-03-11T21:18:50.253381+0000","last_unstale":"2024-03-11T21:19:18.676705+0000","last_undegraded":"2024-03-11T21:19:18.676705+0000","last_fullsized":"2024-03-11T21:19:18.676705+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:10.091469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674232+0000","last_change":"2024-03-11T21:18:54.270313+0000","last_active":"2024-03-11T21:19:18.674232+0000","last_peered":"2024-03-11T21:19:18.674232+0000","last_clean":"2024-03-11T21:19:18.674232+0000","last_became_active":"2024-03-11T21:18:54.270016+0000","last_became_peered":"2024-03-11T21:18:54.270016+0000","last_unstale":"2024-03-11T21:19:18.674232+0000","last_undegraded":"2024-03-11T21:19:18.674232+0000","last_fullsized":"2024-03-11T21:19:18.674232+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:11:30.613393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673962+0000","last_change":"2024-03-11T21:18:52.260631+0000","last_active":"2024-03-11T21:19:18.673962+0000","last_peered":"2024-03-11T21:19:18.673962+0000","last_clean":"2024-03-11T21:19:18.673962+0000","last_became_active":"2024-03-11T21:18:52.260152+0000","last_became_peered":"2024-03-11T21:18:52.260152+0000","last_unstale":"2024-03-11T21:19:18.673962+0000","last_undegraded":"2024-03-11T21:19:18.673962+0000","last_fullsized":"2024-03-11T21:19:18.673962+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:45.743693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674684+0000","last_change":"2024-03-11T21:18:48.248071+0000","last_active":"2024-03-11T21:19:18.674684+0000","last_peered":"2024-03-11T21:19:18.674684+0000","last_clean":"2024-03-11T21:19:18.674684+0000","last_became_active":"2024-03-11T21:18:48.247186+0000","last_became_peered":"2024-03-11T21:18:48.247186+0000","last_unstale":"2024-03-11T21:19:18.674684+0000","last_undegraded":"2024-03-11T21:19:18.674684+0000","last_fullsized":"2024-03-11T21:19:18.674684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:13.309334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675463+0000","last_change":"2024-03-11T21:18:50.254918+0000","last_active":"2024-03-11T21:19:18.675463+0000","last_peered":"2024-03-11T21:19:18.675463+0000","last_clean":"2024-03-11T21:19:18.675463+0000","last_became_active":"2024-03-11T21:18:50.254649+0000","last_became_peered":"2024-03-11T21:18:50.254649+0000","last_unstale":"2024-03-11T21:19:18.675463+0000","last_undegraded":"2024-03-11T21:19:18.675463+0000","last_fullsized":"2024-03-11T21:19:18.675463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:14.847590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675293+0000","last_change":"2024-03-11T21:18:54.269071+0000","last_active":"2024-03-11T21:19:18.675293+0000","last_peered":"2024-03-11T21:19:18.675293+0000","last_clean":"2024-03-11T21:19:18.675293+0000","last_became_active":"2024-03-11T21:18:54.268743+0000","last_became_peered":"2024-03-11T21:18:54.268743+0000","last_unstale":"2024-03-11T21:19:18.675293+0000","last_undegraded":"2024-03-11T21:19:18.675293+0000","last_fullsized":"2024-03-11T21:19:18.675293+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:11:41.549264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296652+0000","last_change":"2024-03-11T21:18:52.262178+0000","last_active":"2024-03-11T21:19:28.296652+0000","last_peered":"2024-03-11T21:19:28.296652+0000","last_clean":"2024-03-11T21:19:28.296652+0000","last_became_active":"2024-03-11T21:18:52.260971+0000","last_became_peered":"2024-03-11T21:18:52.260971+0000","last_unstale":"2024-03-11T21:19:28.296652+0000","last_undegraded":"2024-03-11T21:19:28.296652+0000","last_fullsized":"2024-03-11T21:19:28.296652+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:56:38.859640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674889+0000","last_change":"2024-03-11T21:18:48.250477+0000","last_active":"2024-03-11T21:19:18.674889+0000","last_peered":"2024-03-11T21:19:18.674889+0000","last_clean":"2024-03-11T21:19:18.674889+0000","last_became_active":"2024-03-11T21:18:48.250195+0000","last_became_peered":"2024-03-11T21:18:48.250195+0000","last_unstale":"2024-03-11T21:19:18.674889+0000","last_undegraded":"2024-03-11T21:19:18.674889+0000","last_fullsized":"2024-03-11T21:19:18.674889+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:12:27.924310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674320+0000","last_change":"2024-03-11T21:18:50.252342+0000","last_active":"2024-03-11T21:19:18.674320+0000","last_peered":"2024-03-11T21:19:18.674320+0000","last_clean":"2024-03-11T21:19:18.674320+0000","last_became_active":"2024-03-11T21:18:50.251953+0000","last_became_peered":"2024-03-11T21:18:50.251953+0000","last_unstale":"2024-03-11T21:19:18.674320+0000","last_undegraded":"2024-03-11T21:19:18.674320+0000","last_fullsized":"2024-03-11T21:19:18.674320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:50.910861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674675+0000","last_change":"2024-03-11T21:18:54.271176+0000","last_active":"2024-03-11T21:19:18.674675+0000","last_peered":"2024-03-11T21:19:18.674675+0000","last_clean":"2024-03-11T21:19:18.674675+0000","last_became_active":"2024-03-11T21:18:54.269108+0000","last_became_peered":"2024-03-11T21:18:54.269108+0000","last_unstale":"2024-03-11T21:19:18.674675+0000","last_undegraded":"2024-03-11T21:19:18.674675+0000","last_fullsized":"2024-03-11T21:19:18.674675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:16:06.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296748+0000","last_change":"2024-03-11T21:18:52.261164+0000","last_active":"2024-03-11T21:19:28.296748+0000","last_peered":"2024-03-11T21:19:28.296748+0000","last_clean":"2024-03-11T21:19:28.296748+0000","last_became_active":"2024-03-11T21:18:52.260775+0000","last_became_peered":"2024-03-11T21:18:52.260775+0000","last_unstale":"2024-03-11T21:19:28.296748+0000","last_undegraded":"2024-03-11T21:19:28.296748+0000","last_fullsized":"2024-03-11T21:19:28.296748+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:46.520752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674224+0000","last_change":"2024-03-11T21:18:48.247699+0000","last_active":"2024-03-11T21:19:18.674224+0000","last_peered":"2024-03-11T21:19:18.674224+0000","last_clean":"2024-03-11T21:19:18.674224+0000","last_became_active":"2024-03-11T21:18:48.246931+0000","last_became_peered":"2024-03-11T21:18:48.246931+0000","last_unstale":"2024-03-11T21:19:18.674224+0000","last_undegraded":"2024-03-11T21:19:18.674224+0000","last_fullsized":"2024-03-11T21:19:18.674224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:12:46.448806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675310+0000","last_change":"2024-03-11T21:18:50.253989+0000","last_active":"2024-03-11T21:19:18.675310+0000","last_peered":"2024-03-11T21:19:18.675310+0000","last_clean":"2024-03-11T21:19:18.675310+0000","last_became_active":"2024-03-11T21:18:50.253522+0000","last_became_peered":"2024-03-11T21:18:50.253522+0000","last_unstale":"2024-03-11T21:19:18.675310+0000","last_undegraded":"2024-03-11T21:19:18.675310+0000","last_fullsized":"2024-03-11T21:19:18.675310+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:10.332041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674518+0000","last_change":"2024-03-11T21:18:54.275212+0000","last_active":"2024-03-11T21:19:18.674518+0000","last_peered":"2024-03-11T21:19:18.674518+0000","last_clean":"2024-03-11T21:19:18.674518+0000","last_became_active":"2024-03-11T21:18:54.274808+0000","last_became_peered":"2024-03-11T21:18:54.274808+0000","last_unstale":"2024-03-11T21:19:18.674518+0000","last_undegraded":"2024-03-11T21:19:18.674518+0000","last_fullsized":"2024-03-11T21:19:18.674518+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:04.848457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677472+0000","last_change":"2024-03-11T21:18:52.263762+0000","last_active":"2024-03-11T21:19:18.677472+0000","last_peered":"2024-03-11T21:19:18.677472+0000","last_clean":"2024-03-11T21:19:18.677472+0000","last_became_active":"2024-03-11T21:18:52.262815+0000","last_became_peered":"2024-03-11T21:18:52.262815+0000","last_unstale":"2024-03-11T21:19:18.677472+0000","last_undegraded":"2024-03-11T21:19:18.677472+0000","last_fullsized":"2024-03-11T21:19:18.677472+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:10:20.161132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673730+0000","last_change":"2024-03-11T21:18:48.248664+0000","last_active":"2024-03-11T21:19:18.673730+0000","last_peered":"2024-03-11T21:19:18.673730+0000","last_clean":"2024-03-11T21:19:18.673730+0000","last_became_active":"2024-03-11T21:18:48.247297+0000","last_became_peered":"2024-03-11T21:18:48.247297+0000","last_unstale":"2024-03-11T21:19:18.673730+0000","last_undegraded":"2024-03-11T21:19:18.673730+0000","last_fullsized":"2024-03-11T21:19:18.673730+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:19:11.465458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673350+0000","last_change":"2024-03-11T21:18:50.255500+0000","last_active":"2024-03-11T21:19:18.673350+0000","last_peered":"2024-03-11T21:19:18.673350+0000","last_clean":"2024-03-11T21:19:18.673350+0000","last_became_active":"2024-03-11T21:18:50.255157+0000","last_became_peered":"2024-03-11T21:18:50.255157+0000","last_unstale":"2024-03-11T21:19:18.673350+0000","last_undegraded":"2024-03-11T21:19:18.673350+0000","last_fullsized":"2024-03-11T21:19:18.673350+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:43.208883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677329+0000","last_change":"2024-03-11T21:18:54.271440+0000","last_active":"2024-03-11T21:19:18.677329+0000","last_peered":"2024-03-11T21:19:18.677329+0000","last_clean":"2024-03-11T21:19:18.677329+0000","last_became_active":"2024-03-11T21:18:54.271145+0000","last_became_peered":"2024-03-11T21:18:54.271145+0000","last_unstale":"2024-03-11T21:19:18.677329+0000","last_undegraded":"2024-03-11T21:19:18.677329+0000","last_fullsized":"2024-03-11T21:19:18.677329+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:58:10.480545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676434+0000","last_change":"2024-03-11T21:18:52.263749+0000","last_active":"2024-03-11T21:19:18.676434+0000","last_peered":"2024-03-11T21:19:18.676434+0000","last_clean":"2024-03-11T21:19:18.676434+0000","last_became_active":"2024-03-11T21:18:52.263403+0000","last_became_peered":"2024-03-11T21:18:52.263403+0000","last_unstale":"2024-03-11T21:19:18.676434+0000","last_undegraded":"2024-03-11T21:19:18.676434+0000","last_fullsized":"2024-03-11T21:19:18.676434+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:56:59.967077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673651+0000","last_change":"2024-03-11T21:18:48.248996+0000","last_active":"2024-03-11T21:19:18.673651+0000","last_peered":"2024-03-11T21:19:18.673651+0000","last_clean":"2024-03-11T21:19:18.673651+0000","last_became_active":"2024-03-11T21:18:48.248697+0000","last_became_peered":"2024-03-11T21:18:48.248697+0000","last_unstale":"2024-03-11T21:19:18.673651+0000","last_undegraded":"2024-03-11T21:19:18.673651+0000","last_fullsized":"2024-03-11T21:19:18.673651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:29.290103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673856+0000","last_change":"2024-03-11T21:18:50.259514+0000","last_active":"2024-03-11T21:19:18.673856+0000","last_peered":"2024-03-11T21:19:18.673856+0000","last_clean":"2024-03-11T21:19:18.673856+0000","last_became_active":"2024-03-11T21:18:50.259040+0000","last_became_peered":"2024-03-11T21:18:50.259040+0000","last_unstale":"2024-03-11T21:19:18.673856+0000","last_undegraded":"2024-03-11T21:19:18.673856+0000","last_fullsized":"2024-03-11T21:19:18.673856+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:13:51.913186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674549+0000","last_change":"2024-03-11T21:18:54.269923+0000","last_active":"2024-03-11T21:19:18.674549+0000","last_peered":"2024-03-11T21:19:18.674549+0000","last_clean":"2024-03-11T21:19:18.674549+0000","last_became_active":"2024-03-11T21:18:54.269569+0000","last_became_peered":"2024-03-11T21:18:54.269569+0000","last_unstale":"2024-03-11T21:19:18.674549+0000","last_undegraded":"2024-03-11T21:19:18.674549+0000","last_fullsized":"2024-03-11T21:19:18.674549+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:42:01.506214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676698+0000","last_change":"2024-03-11T21:18:48.247869+0000","last_active":"2024-03-11T21:19:18.676698+0000","last_peered":"2024-03-11T21:19:18.676698+0000","last_clean":"2024-03-11T21:19:18.676698+0000","last_became_active":"2024-03-11T21:18:48.247594+0000","last_became_peered":"2024-03-11T21:18:48.247594+0000","last_unstale":"2024-03-11T21:19:18.676698+0000","last_undegraded":"2024-03-11T21:19:18.676698+0000","last_fullsized":"2024-03-11T21:19:18.676698+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:57:53.797906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676522+0000","last_change":"2024-03-11T21:18:50.254338+0000","last_active":"2024-03-11T21:19:18.676522+0000","last_peered":"2024-03-11T21:19:18.676522+0000","last_clean":"2024-03-11T21:19:18.676522+0000","last_became_active":"2024-03-11T21:18:50.254086+0000","last_became_peered":"2024-03-11T21:18:50.254086+0000","last_unstale":"2024-03-11T21:19:18.676522+0000","last_undegraded":"2024-03-11T21:19:18.676522+0000","last_fullsized":"2024-03-11T21:19:18.676522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:30.941114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296736+0000","last_change":"2024-03-11T21:18:52.262882+0000","last_active":"2024-03-11T21:19:28.296736+0000","last_peered":"2024-03-11T21:19:28.296736+0000","last_clean":"2024-03-11T21:19:28.296736+0000","last_became_active":"2024-03-11T21:18:52.261840+0000","last_became_peered":"2024-03-11T21:18:52.261840+0000","last_unstale":"2024-03-11T21:19:28.296736+0000","last_undegraded":"2024-03-11T21:19:28.296736+0000","last_fullsized":"2024-03-11T21:19:28.296736+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:00.609082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.672860+0000","last_change":"2024-03-11T21:18:54.267931+0000","last_active":"2024-03-11T21:19:18.672860+0000","last_peered":"2024-03-11T21:19:18.672860+0000","last_clean":"2024-03-11T21:19:18.672860+0000","last_became_active":"2024-03-11T21:18:54.267399+0000","last_became_peered":"2024-03-11T21:18:54.267399+0000","last_unstale":"2024-03-11T21:19:18.672860+0000","last_undegraded":"2024-03-11T21:19:18.672860+0000","last_fullsized":"2024-03-11T21:19:18.672860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:42:22.105023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675924+0000","last_change":"2024-03-11T21:18:48.245449+0000","last_active":"2024-03-11T21:19:18.675924+0000","last_peered":"2024-03-11T21:19:18.675924+0000","last_clean":"2024-03-11T21:19:18.675924+0000","last_became_active":"2024-03-11T21:18:48.245109+0000","last_became_peered":"2024-03-11T21:18:48.245109+0000","last_unstale":"2024-03-11T21:19:18.675924+0000","last_undegraded":"2024-03-11T21:19:18.675924+0000","last_fullsized":"2024-03-11T21:19:18.675924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:01.506661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296552+0000","last_change":"2024-03-11T21:18:50.255542+0000","last_active":"2024-03-11T21:19:28.296552+0000","last_peered":"2024-03-11T21:19:28.296552+0000","last_clean":"2024-03-11T21:19:28.296552+0000","last_became_active":"2024-03-11T21:18:50.255201+0000","last_became_peered":"2024-03-11T21:18:50.255201+0000","last_unstale":"2024-03-11T21:19:28.296552+0000","last_undegraded":"2024-03-11T21:19:28.296552+0000","last_fullsized":"2024-03-11T21:19:28.296552+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:12:45.241972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673990+0000","last_change":"2024-03-11T21:18:52.255738+0000","last_active":"2024-03-11T21:19:18.673990+0000","last_peered":"2024-03-11T21:19:18.673990+0000","last_clean":"2024-03-11T21:19:18.673990+0000","last_became_active":"2024-03-11T21:18:52.255512+0000","last_became_peered":"2024-03-11T21:18:52.255512+0000","last_unstale":"2024-03-11T21:19:18.673990+0000","last_undegraded":"2024-03-11T21:19:18.673990+0000","last_fullsized":"2024-03-11T21:19:18.673990+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:56.558089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675068+0000","last_change":"2024-03-11T21:18:54.269672+0000","last_active":"2024-03-11T21:19:18.675068+0000","last_peered":"2024-03-11T21:19:18.675068+0000","last_clean":"2024-03-11T21:19:18.675068+0000","last_became_active":"2024-03-11T21:18:54.269345+0000","last_became_peered":"2024-03-11T21:18:54.269345+0000","last_unstale":"2024-03-11T21:19:18.675068+0000","last_undegraded":"2024-03-11T21:19:18.675068+0000","last_fullsized":"2024-03-11T21:19:18.675068+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:20:47.364834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674335+0000","last_change":"2024-03-11T21:18:48.247179+0000","last_active":"2024-03-11T21:19:18.674335+0000","last_peered":"2024-03-11T21:19:18.674335+0000","last_clean":"2024-03-11T21:19:18.674335+0000","last_became_active":"2024-03-11T21:18:48.246841+0000","last_became_peered":"2024-03-11T21:18:48.246841+0000","last_unstale":"2024-03-11T21:19:18.674335+0000","last_undegraded":"2024-03-11T21:19:18.674335+0000","last_fullsized":"2024-03-11T21:19:18.674335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:33:39.174507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675476+0000","last_change":"2024-03-11T21:18:50.253332+0000","last_active":"2024-03-11T21:19:18.675476+0000","last_peered":"2024-03-11T21:19:18.675476+0000","last_clean":"2024-03-11T21:19:18.675476+0000","last_became_active":"2024-03-11T21:18:50.252333+0000","last_became_peered":"2024-03-11T21:18:50.252333+0000","last_unstale":"2024-03-11T21:19:18.675476+0000","last_undegraded":"2024-03-11T21:19:18.675476+0000","last_fullsized":"2024-03-11T21:19:18.675476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:31.270474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675387+0000","last_change":"2024-03-11T21:18:52.263625+0000","last_active":"2024-03-11T21:19:18.675387+0000","last_peered":"2024-03-11T21:19:18.675387+0000","last_clean":"2024-03-11T21:19:18.675387+0000","last_became_active":"2024-03-11T21:18:52.263227+0000","last_became_peered":"2024-03-11T21:18:52.263227+0000","last_unstale":"2024-03-11T21:19:18.675387+0000","last_undegraded":"2024-03-11T21:19:18.675387+0000","last_fullsized":"2024-03-11T21:19:18.675387+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:41:04.484588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677170+0000","last_change":"2024-03-11T21:18:54.271572+0000","last_active":"2024-03-11T21:19:18.677170+0000","last_peered":"2024-03-11T21:19:18.677170+0000","last_clean":"2024-03-11T21:19:18.677170+0000","last_became_active":"2024-03-11T21:18:54.271240+0000","last_became_peered":"2024-03-11T21:18:54.271240+0000","last_unstale":"2024-03-11T21:19:18.677170+0000","last_undegraded":"2024-03-11T21:19:18.677170+0000","last_fullsized":"2024-03-11T21:19:18.677170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:22.422794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673399+0000","last_change":"2024-03-11T21:18:50.259525+0000","last_active":"2024-03-11T21:19:18.673399+0000","last_peered":"2024-03-11T21:19:18.673399+0000","last_clean":"2024-03-11T21:19:18.673399+0000","last_became_active":"2024-03-11T21:18:50.259248+0000","last_became_peered":"2024-03-11T21:18:50.259248+0000","last_unstale":"2024-03-11T21:19:18.673399+0000","last_undegraded":"2024-03-11T21:19:18.673399+0000","last_fullsized":"2024-03-11T21:19:18.673399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:08.756866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673125+0000","last_change":"2024-03-11T21:18:48.245861+0000","last_active":"2024-03-11T21:19:18.673125+0000","last_peered":"2024-03-11T21:19:18.673125+0000","last_clean":"2024-03-11T21:19:18.673125+0000","last_became_active":"2024-03-11T21:18:48.245351+0000","last_became_peered":"2024-03-11T21:18:48.245351+0000","last_unstale":"2024-03-11T21:19:18.673125+0000","last_undegraded":"2024-03-11T21:19:18.673125+0000","last_fullsized":"2024-03-11T21:19:18.673125+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:47:55.849112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677622+0000","last_change":"2024-03-11T21:18:52.264231+0000","last_active":"2024-03-11T21:19:18.677622+0000","last_peered":"2024-03-11T21:19:18.677622+0000","last_clean":"2024-03-11T21:19:18.677622+0000","last_became_active":"2024-03-11T21:18:52.263604+0000","last_became_peered":"2024-03-11T21:18:52.263604+0000","last_unstale":"2024-03-11T21:19:18.677622+0000","last_undegraded":"2024-03-11T21:19:18.677622+0000","last_fullsized":"2024-03-11T21:19:18.677622+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:37:01.641209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675483+0000","last_change":"2024-03-11T21:18:54.270112+0000","last_active":"2024-03-11T21:19:18.675483+0000","last_peered":"2024-03-11T21:19:18.675483+0000","last_clean":"2024-03-11T21:19:18.675483+0000","last_became_active":"2024-03-11T21:18:54.269793+0000","last_became_peered":"2024-03-11T21:18:54.269793+0000","last_unstale":"2024-03-11T21:19:18.675483+0000","last_undegraded":"2024-03-11T21:19:18.675483+0000","last_fullsized":"2024-03-11T21:19:18.675483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:36.132338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674120+0000","last_change":"2024-03-11T21:18:50.252671+0000","last_active":"2024-03-11T21:19:18.674120+0000","last_peered":"2024-03-11T21:19:18.674120+0000","last_clean":"2024-03-11T21:19:18.674120+0000","last_became_active":"2024-03-11T21:18:50.252279+0000","last_became_peered":"2024-03-11T21:18:50.252279+0000","last_unstale":"2024-03-11T21:19:18.674120+0000","last_undegraded":"2024-03-11T21:19:18.674120+0000","last_fullsized":"2024-03-11T21:19:18.674120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:47:01.954334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674416+0000","last_change":"2024-03-11T21:18:48.247627+0000","last_active":"2024-03-11T21:19:18.674416+0000","last_peered":"2024-03-11T21:19:18.674416+0000","last_clean":"2024-03-11T21:19:18.674416+0000","last_became_active":"2024-03-11T21:18:48.247095+0000","last_became_peered":"2024-03-11T21:18:48.247095+0000","last_unstale":"2024-03-11T21:19:18.674416+0000","last_undegraded":"2024-03-11T21:19:18.674416+0000","last_fullsized":"2024-03-11T21:19:18.674416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:07:13.692264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:20.769805+0000","last_change":"2024-03-11T21:18:37.228580+0000","last_active":"2024-03-11T21:19:20.769805+0000","last_peered":"2024-03-11T21:19:20.769805+0000","last_clean":"2024-03-11T21:19:20.769805+0000","last_became_active":"2024-03-11T21:18:37.228208+0000","last_became_peered":"2024-03-11T21:18:37.228208+0000","last_unstale":"2024-03-11T21:19:20.769805+0000","last_undegraded":"2024-03-11T21:19:20.769805+0000","last_fullsized":"2024-03-11T21:19:20.769805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:36.216238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:36.216238+0000","last_clean_scrub_stamp":"2024-03-11T21:18:36.216238+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:55.801174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675745+0000","last_change":"2024-03-11T21:18:52.257644+0000","last_active":"2024-03-11T21:19:18.675745+0000","last_peered":"2024-03-11T21:19:18.675745+0000","last_clean":"2024-03-11T21:19:18.675745+0000","last_became_active":"2024-03-11T21:18:52.257204+0000","last_became_peered":"2024-03-11T21:18:52.257204+0000","last_unstale":"2024-03-11T21:19:18.675745+0000","last_undegraded":"2024-03-11T21:19:18.675745+0000","last_fullsized":"2024-03-11T21:19:18.675745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:44:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674725+0000","last_change":"2024-03-11T21:18:54.272781+0000","last_active":"2024-03-11T21:19:18.674725+0000","last_peered":"2024-03-11T21:19:18.674725+0000","last_clean":"2024-03-11T21:19:18.674725+0000","last_became_active":"2024-03-11T21:18:54.272442+0000","last_became_peered":"2024-03-11T21:18:54.272442+0000","last_unstale":"2024-03-11T21:19:18.674725+0000","last_undegraded":"2024-03-11T21:19:18.674725+0000","last_fullsized":"2024-03-11T21:19:18.674725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:06:31.016510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673479+0000","last_change":"2024-03-11T21:18:50.256073+0000","last_active":"2024-03-11T21:19:18.673479+0000","last_peered":"2024-03-11T21:19:18.673479+0000","last_clean":"2024-03-11T21:19:18.673479+0000","last_became_active":"2024-03-11T21:18:50.253649+0000","last_became_peered":"2024-03-11T21:18:50.253649+0000","last_unstale":"2024-03-11T21:19:18.673479+0000","last_undegraded":"2024-03-11T21:19:18.673479+0000","last_fullsized":"2024-03-11T21:19:18.673479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:18:54.215008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673557+0000","last_change":"2024-03-11T21:18:48.250192+0000","last_active":"2024-03-11T21:19:18.673557+0000","last_peered":"2024-03-11T21:19:18.673557+0000","last_clean":"2024-03-11T21:19:18.673557+0000","last_became_active":"2024-03-11T21:18:48.249808+0000","last_became_peered":"2024-03-11T21:18:48.249808+0000","last_unstale":"2024-03-11T21:19:18.673557+0000","last_undegraded":"2024-03-11T21:19:18.673557+0000","last_fullsized":"2024-03-11T21:19:18.673557+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:33:00.890001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675640+0000","last_change":"2024-03-11T21:18:52.262568+0000","last_active":"2024-03-11T21:19:18.675640+0000","last_peered":"2024-03-11T21:19:18.675640+0000","last_clean":"2024-03-11T21:19:18.675640+0000","last_became_active":"2024-03-11T21:18:52.262262+0000","last_became_peered":"2024-03-11T21:18:52.262262+0000","last_unstale":"2024-03-11T21:19:18.675640+0000","last_undegraded":"2024-03-11T21:19:18.675640+0000","last_fullsized":"2024-03-11T21:19:18.675640+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:05:19.605432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674073+0000","last_change":"2024-03-11T21:18:54.269037+0000","last_active":"2024-03-11T21:19:18.674073+0000","last_peered":"2024-03-11T21:19:18.674073+0000","last_clean":"2024-03-11T21:19:18.674073+0000","last_became_active":"2024-03-11T21:18:54.268275+0000","last_became_peered":"2024-03-11T21:18:54.268275+0000","last_unstale":"2024-03-11T21:19:18.674073+0000","last_undegraded":"2024-03-11T21:19:18.674073+0000","last_fullsized":"2024-03-11T21:19:18.674073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:42:01.189789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675554+0000","last_change":"2024-03-11T21:18:48.248663+0000","last_active":"2024-03-11T21:19:18.675554+0000","last_peered":"2024-03-11T21:19:18.675554+0000","last_clean":"2024-03-11T21:19:18.675554+0000","last_became_active":"2024-03-11T21:18:48.248376+0000","last_became_peered":"2024-03-11T21:18:48.248376+0000","last_unstale":"2024-03-11T21:19:18.675554+0000","last_undegraded":"2024-03-11T21:19:18.675554+0000","last_fullsized":"2024-03-11T21:19:18.675554+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:08:26.780594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":81,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296291+0000","last_change":"2024-03-11T21:18:50.253683+0000","last_active":"2024-03-11T21:19:28.296291+0000","last_peered":"2024-03-11T21:19:28.296291+0000","last_clean":"2024-03-11T21:19:28.296291+0000","last_became_active":"2024-03-11T21:18:50.253244+0000","last_became_peered":"2024-03-11T21:18:50.253244+0000","last_unstale":"2024-03-11T21:19:28.296291+0000","last_undegraded":"2024-03-11T21:19:28.296291+0000","last_fullsized":"2024-03-11T21:19:28.296291+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:46:42.254634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296702+0000","last_change":"2024-03-11T21:18:52.261066+0000","last_active":"2024-03-11T21:19:28.296702+0000","last_peered":"2024-03-11T21:19:28.296702+0000","last_clean":"2024-03-11T21:19:28.296702+0000","last_became_active":"2024-03-11T21:18:52.260664+0000","last_became_peered":"2024-03-11T21:18:52.260664+0000","last_unstale":"2024-03-11T21:19:28.296702+0000","last_undegraded":"2024-03-11T21:19:28.296702+0000","last_fullsized":"2024-03-11T21:19:28.296702+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:53:05.029022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673263+0000","last_change":"2024-03-11T21:18:54.271138+0000","last_active":"2024-03-11T21:19:18.673263+0000","last_peered":"2024-03-11T21:19:18.673263+0000","last_clean":"2024-03-11T21:19:18.673263+0000","last_became_active":"2024-03-11T21:18:54.270413+0000","last_became_peered":"2024-03-11T21:18:54.270413+0000","last_unstale":"2024-03-11T21:19:18.673263+0000","last_undegraded":"2024-03-11T21:19:18.673263+0000","last_fullsized":"2024-03-11T21:19:18.673263+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:46:17.724950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677139+0000","last_change":"2024-03-11T21:18:48.246093+0000","last_active":"2024-03-11T21:19:18.677139+0000","last_peered":"2024-03-11T21:19:18.677139+0000","last_clean":"2024-03-11T21:19:18.677139+0000","last_became_active":"2024-03-11T21:18:48.245628+0000","last_became_peered":"2024-03-11T21:18:48.245628+0000","last_unstale":"2024-03-11T21:19:18.677139+0000","last_undegraded":"2024-03-11T21:19:18.677139+0000","last_fullsized":"2024-03-11T21:19:18.677139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:09:46.690889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.678030+0000","last_change":"2024-03-11T21:18:50.258139+0000","last_active":"2024-03-11T21:19:18.678030+0000","last_peered":"2024-03-11T21:19:18.678030+0000","last_clean":"2024-03-11T21:19:18.678030+0000","last_became_active":"2024-03-11T21:18:50.256723+0000","last_became_peered":"2024-03-11T21:18:50.256723+0000","last_unstale":"2024-03-11T21:19:18.678030+0000","last_undegraded":"2024-03-11T21:19:18.678030+0000","last_fullsized":"2024-03-11T21:19:18.678030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:17:32.240855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296283+0000","last_change":"2024-03-11T21:18:52.259011+0000","last_active":"2024-03-11T21:19:28.296283+0000","last_peered":"2024-03-11T21:19:28.296283+0000","last_clean":"2024-03-11T21:19:28.296283+0000","last_became_active":"2024-03-11T21:18:52.258629+0000","last_became_peered":"2024-03-11T21:18:52.258629+0000","last_unstale":"2024-03-11T21:19:28.296283+0000","last_undegraded":"2024-03-11T21:19:28.296283+0000","last_fullsized":"2024-03-11T21:19:28.296283+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:53.726500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674492+0000","last_change":"2024-03-11T21:18:54.269645+0000","last_active":"2024-03-11T21:19:18.674492+0000","last_peered":"2024-03-11T21:19:18.674492+0000","last_clean":"2024-03-11T21:19:18.674492+0000","last_became_active":"2024-03-11T21:18:54.269306+0000","last_became_peered":"2024-03-11T21:18:54.269306+0000","last_unstale":"2024-03-11T21:19:18.674492+0000","last_undegraded":"2024-03-11T21:19:18.674492+0000","last_fullsized":"2024-03-11T21:19:18.674492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:02:56.987598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675813+0000","last_change":"2024-03-11T21:18:48.249851+0000","last_active":"2024-03-11T21:19:18.675813+0000","last_peered":"2024-03-11T21:19:18.675813+0000","last_clean":"2024-03-11T21:19:18.675813+0000","last_became_active":"2024-03-11T21:18:48.249511+0000","last_became_peered":"2024-03-11T21:18:48.249511+0000","last_unstale":"2024-03-11T21:19:18.675813+0000","last_undegraded":"2024-03-11T21:19:18.675813+0000","last_fullsized":"2024-03-11T21:19:18.675813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:12.080235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676576+0000","last_change":"2024-03-11T21:18:50.253791+0000","last_active":"2024-03-11T21:19:18.676576+0000","last_peered":"2024-03-11T21:19:18.676576+0000","last_clean":"2024-03-11T21:19:18.676576+0000","last_became_active":"2024-03-11T21:18:50.252999+0000","last_became_peered":"2024-03-11T21:18:50.252999+0000","last_unstale":"2024-03-11T21:19:18.676576+0000","last_undegraded":"2024-03-11T21:19:18.676576+0000","last_fullsized":"2024-03-11T21:19:18.676576+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:46.092445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673665+0000","last_change":"2024-03-11T21:18:52.262895+0000","last_active":"2024-03-11T21:19:18.673665+0000","last_peered":"2024-03-11T21:19:18.673665+0000","last_clean":"2024-03-11T21:19:18.673665+0000","last_became_active":"2024-03-11T21:18:52.261884+0000","last_became_peered":"2024-03-11T21:18:52.261884+0000","last_unstale":"2024-03-11T21:19:18.673665+0000","last_undegraded":"2024-03-11T21:19:18.673665+0000","last_fullsized":"2024-03-11T21:19:18.673665+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:01:24.568625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675402+0000","last_change":"2024-03-11T21:18:54.271306+0000","last_active":"2024-03-11T21:19:18.675402+0000","last_peered":"2024-03-11T21:19:18.675402+0000","last_clean":"2024-03-11T21:19:18.675402+0000","last_became_active":"2024-03-11T21:18:54.270132+0000","last_became_peered":"2024-03-11T21:18:54.270132+0000","last_unstale":"2024-03-11T21:19:18.675402+0000","last_undegraded":"2024-03-11T21:19:18.675402+0000","last_fullsized":"2024-03-11T21:19:18.675402+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:48.135906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296424+0000","last_change":"2024-03-11T21:18:52.258699+0000","last_active":"2024-03-11T21:19:28.296424+0000","last_peered":"2024-03-11T21:19:28.296424+0000","last_clean":"2024-03-11T21:19:28.296424+0000","last_became_active":"2024-03-11T21:18:52.257886+0000","last_became_peered":"2024-03-11T21:18:52.257886+0000","last_unstale":"2024-03-11T21:19:28.296424+0000","last_undegraded":"2024-03-11T21:19:28.296424+0000","last_fullsized":"2024-03-11T21:19:28.296424+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:09.010264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673571+0000","last_change":"2024-03-11T21:18:48.250174+0000","last_active":"2024-03-11T21:19:18.673571+0000","last_peered":"2024-03-11T21:19:18.673571+0000","last_clean":"2024-03-11T21:19:18.673571+0000","last_became_active":"2024-03-11T21:18:48.249814+0000","last_became_peered":"2024-03-11T21:18:48.249814+0000","last_unstale":"2024-03-11T21:19:18.673571+0000","last_undegraded":"2024-03-11T21:19:18.673571+0000","last_fullsized":"2024-03-11T21:19:18.673571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:08.168756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674077+0000","last_change":"2024-03-11T21:18:50.259280+0000","last_active":"2024-03-11T21:19:18.674077+0000","last_peered":"2024-03-11T21:19:18.674077+0000","last_clean":"2024-03-11T21:19:18.674077+0000","last_became_active":"2024-03-11T21:18:50.258990+0000","last_became_peered":"2024-03-11T21:18:50.258990+0000","last_unstale":"2024-03-11T21:19:18.674077+0000","last_undegraded":"2024-03-11T21:19:18.674077+0000","last_fullsized":"2024-03-11T21:19:18.674077+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:36:06.935178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675733+0000","last_change":"2024-03-11T21:18:54.272376+0000","last_active":"2024-03-11T21:19:18.675733+0000","last_peered":"2024-03-11T21:19:18.675733+0000","last_clean":"2024-03-11T21:19:18.675733+0000","last_became_active":"2024-03-11T21:18:54.272053+0000","last_became_peered":"2024-03-11T21:18:54.272053+0000","last_unstale":"2024-03-11T21:19:18.675733+0000","last_undegraded":"2024-03-11T21:19:18.675733+0000","last_fullsized":"2024-03-11T21:19:18.675733+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:27:34.884608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.677518+0000","last_change":"2024-03-11T21:18:52.263997+0000","last_active":"2024-03-11T21:19:18.677518+0000","last_peered":"2024-03-11T21:19:18.677518+0000","last_clean":"2024-03-11T21:19:18.677518+0000","last_became_active":"2024-03-11T21:18:52.263016+0000","last_became_peered":"2024-03-11T21:18:52.263016+0000","last_unstale":"2024-03-11T21:19:18.677518+0000","last_undegraded":"2024-03-11T21:19:18.677518+0000","last_fullsized":"2024-03-11T21:19:18.677518+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:48.586572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673322+0000","last_change":"2024-03-11T21:18:48.244043+0000","last_active":"2024-03-11T21:19:18.673322+0000","last_peered":"2024-03-11T21:19:18.673322+0000","last_clean":"2024-03-11T21:19:18.673322+0000","last_became_active":"2024-03-11T21:18:48.243558+0000","last_became_peered":"2024-03-11T21:18:48.243558+0000","last_unstale":"2024-03-11T21:19:18.673322+0000","last_undegraded":"2024-03-11T21:19:18.673322+0000","last_fullsized":"2024-03-11T21:19:18.673322+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:04.521470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673777+0000","last_change":"2024-03-11T21:18:50.254872+0000","last_active":"2024-03-11T21:19:18.673777+0000","last_peered":"2024-03-11T21:19:18.673777+0000","last_clean":"2024-03-11T21:19:18.673777+0000","last_became_active":"2024-03-11T21:18:50.254569+0000","last_became_peered":"2024-03-11T21:18:50.254569+0000","last_unstale":"2024-03-11T21:19:18.673777+0000","last_undegraded":"2024-03-11T21:19:18.673777+0000","last_fullsized":"2024-03-11T21:19:18.673777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:05:47.625061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675049+0000","last_change":"2024-03-11T21:18:54.270024+0000","last_active":"2024-03-11T21:19:18.675049+0000","last_peered":"2024-03-11T21:19:18.675049+0000","last_clean":"2024-03-11T21:19:18.675049+0000","last_became_active":"2024-03-11T21:18:54.269724+0000","last_became_peered":"2024-03-11T21:18:54.269724+0000","last_unstale":"2024-03-11T21:19:18.675049+0000","last_undegraded":"2024-03-11T21:19:18.675049+0000","last_fullsized":"2024-03-11T21:19:18.675049+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:04:58.156964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674559+0000","last_change":"2024-03-11T21:18:48.247181+0000","last_active":"2024-03-11T21:19:18.674559+0000","last_peered":"2024-03-11T21:19:18.674559+0000","last_clean":"2024-03-11T21:19:18.674559+0000","last_became_active":"2024-03-11T21:18:48.246859+0000","last_became_peered":"2024-03-11T21:18:48.246859+0000","last_unstale":"2024-03-11T21:19:18.674559+0000","last_undegraded":"2024-03-11T21:19:18.674559+0000","last_fullsized":"2024-03-11T21:19:18.674559+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:53.677395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673374+0000","last_change":"2024-03-11T21:18:50.253471+0000","last_active":"2024-03-11T21:19:18.673374+0000","last_peered":"2024-03-11T21:19:18.673374+0000","last_clean":"2024-03-11T21:19:18.673374+0000","last_became_active":"2024-03-11T21:18:50.252846+0000","last_became_peered":"2024-03-11T21:18:50.252846+0000","last_unstale":"2024-03-11T21:19:18.673374+0000","last_undegraded":"2024-03-11T21:19:18.673374+0000","last_fullsized":"2024-03-11T21:19:18.673374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:25:40.921942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674433+0000","last_change":"2024-03-11T21:18:52.259569+0000","last_active":"2024-03-11T21:19:18.674433+0000","last_peered":"2024-03-11T21:19:18.674433+0000","last_clean":"2024-03-11T21:19:18.674433+0000","last_became_active":"2024-03-11T21:18:52.259095+0000","last_became_peered":"2024-03-11T21:18:52.259095+0000","last_unstale":"2024-03-11T21:19:18.674433+0000","last_undegraded":"2024-03-11T21:19:18.674433+0000","last_fullsized":"2024-03-11T21:19:18.674433+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:01.981684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674986+0000","last_change":"2024-03-11T21:18:54.270644+0000","last_active":"2024-03-11T21:19:18.674986+0000","last_peered":"2024-03-11T21:19:18.674986+0000","last_clean":"2024-03-11T21:19:18.674986+0000","last_became_active":"2024-03-11T21:18:54.270107+0000","last_became_peered":"2024-03-11T21:18:54.270107+0000","last_unstale":"2024-03-11T21:19:18.674986+0000","last_undegraded":"2024-03-11T21:19:18.674986+0000","last_fullsized":"2024-03-11T21:19:18.674986+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:13.964645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:28.296425+0000","last_change":"2024-03-11T21:18:52.261525+0000","last_active":"2024-03-11T21:19:28.296425+0000","last_peered":"2024-03-11T21:19:28.296425+0000","last_clean":"2024-03-11T21:19:28.296425+0000","last_became_active":"2024-03-11T21:18:52.261203+0000","last_became_peered":"2024-03-11T21:18:52.261203+0000","last_unstale":"2024-03-11T21:19:28.296425+0000","last_undegraded":"2024-03-11T21:19:28.296425+0000","last_fullsized":"2024-03-11T21:19:28.296425+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:59:52.473583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674745+0000","last_change":"2024-03-11T21:18:48.250700+0000","last_active":"2024-03-11T21:19:18.674745+0000","last_peered":"2024-03-11T21:19:18.674745+0000","last_clean":"2024-03-11T21:19:18.674745+0000","last_became_active":"2024-03-11T21:18:48.250248+0000","last_became_peered":"2024-03-11T21:18:48.250248+0000","last_unstale":"2024-03-11T21:19:18.674745+0000","last_undegraded":"2024-03-11T21:19:18.674745+0000","last_fullsized":"2024-03-11T21:19:18.674745+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:45.525049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675242+0000","last_change":"2024-03-11T21:18:50.253308+0000","last_active":"2024-03-11T21:19:18.675242+0000","last_peered":"2024-03-11T21:19:18.675242+0000","last_clean":"2024-03-11T21:19:18.675242+0000","last_became_active":"2024-03-11T21:18:50.251849+0000","last_became_peered":"2024-03-11T21:18:50.251849+0000","last_unstale":"2024-03-11T21:19:18.675242+0000","last_undegraded":"2024-03-11T21:19:18.675242+0000","last_fullsized":"2024-03-11T21:19:18.675242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:44:03.848772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.674527+0000","last_change":"2024-03-11T21:18:54.272927+0000","last_active":"2024-03-11T21:19:18.674527+0000","last_peered":"2024-03-11T21:19:18.674527+0000","last_clean":"2024-03-11T21:19:18.674527+0000","last_became_active":"2024-03-11T21:18:54.272552+0000","last_became_peered":"2024-03-11T21:18:54.272552+0000","last_unstale":"2024-03-11T21:19:18.674527+0000","last_undegraded":"2024-03-11T21:19:18.674527+0000","last_fullsized":"2024-03-11T21:19:18.674527+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:33:47.842278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676257+0000","last_change":"2024-03-11T21:18:48.247121+0000","last_active":"2024-03-11T21:19:18.676257+0000","last_peered":"2024-03-11T21:19:18.676257+0000","last_clean":"2024-03-11T21:19:18.676257+0000","last_became_active":"2024-03-11T21:18:48.246827+0000","last_became_peered":"2024-03-11T21:18:48.246827+0000","last_unstale":"2024-03-11T21:19:18.676257+0000","last_undegraded":"2024-03-11T21:19:18.676257+0000","last_fullsized":"2024-03-11T21:19:18.676257+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:06.460925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676114+0000","last_change":"2024-03-11T21:18:50.255062+0000","last_active":"2024-03-11T21:19:18.676114+0000","last_peered":"2024-03-11T21:19:18.676114+0000","last_clean":"2024-03-11T21:19:18.676114+0000","last_became_active":"2024-03-11T21:18:50.253579+0000","last_became_peered":"2024-03-11T21:18:50.253579+0000","last_unstale":"2024-03-11T21:19:18.676114+0000","last_undegraded":"2024-03-11T21:19:18.676114+0000","last_fullsized":"2024-03-11T21:19:18.676114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:48:23.883265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673722+0000","last_change":"2024-03-11T21:18:52.257009+0000","last_active":"2024-03-11T21:19:18.673722+0000","last_peered":"2024-03-11T21:19:18.673722+0000","last_clean":"2024-03-11T21:19:18.673722+0000","last_became_active":"2024-03-11T21:18:52.256109+0000","last_became_peered":"2024-03-11T21:18:52.256109+0000","last_unstale":"2024-03-11T21:19:18.673722+0000","last_undegraded":"2024-03-11T21:19:18.673722+0000","last_fullsized":"2024-03-11T21:19:18.673722+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:49:52.715019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673580+0000","last_change":"2024-03-11T21:18:54.268744+0000","last_active":"2024-03-11T21:19:18.673580+0000","last_peered":"2024-03-11T21:19:18.673580+0000","last_clean":"2024-03-11T21:19:18.673580+0000","last_became_active":"2024-03-11T21:18:54.267967+0000","last_became_peered":"2024-03-11T21:18:54.267967+0000","last_unstale":"2024-03-11T21:19:18.673580+0000","last_undegraded":"2024-03-11T21:19:18.673580+0000","last_fullsized":"2024-03-11T21:19:18.673580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:24:23.737447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675364+0000","last_change":"2024-03-11T21:18:48.250517+0000","last_active":"2024-03-11T21:19:18.675364+0000","last_peered":"2024-03-11T21:19:18.675364+0000","last_clean":"2024-03-11T21:19:18.675364+0000","last_became_active":"2024-03-11T21:18:48.250220+0000","last_became_peered":"2024-03-11T21:18:48.250220+0000","last_unstale":"2024-03-11T21:19:18.675364+0000","last_undegraded":"2024-03-11T21:19:18.675364+0000","last_fullsized":"2024-03-11T21:19:18.675364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:25:15.099637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675267+0000","last_change":"2024-03-11T21:18:50.256216+0000","last_active":"2024-03-11T21:19:18.675267+0000","last_peered":"2024-03-11T21:19:18.675267+0000","last_clean":"2024-03-11T21:19:18.675267+0000","last_became_active":"2024-03-11T21:18:50.255795+0000","last_became_peered":"2024-03-11T21:18:50.255795+0000","last_unstale":"2024-03-11T21:19:18.675267+0000","last_undegraded":"2024-03-11T21:19:18.675267+0000","last_fullsized":"2024-03-11T21:19:18.675267+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:30:08.862808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673122+0000","last_change":"2024-03-11T21:18:52.259482+0000","last_active":"2024-03-11T21:19:18.673122+0000","last_peered":"2024-03-11T21:19:18.673122+0000","last_clean":"2024-03-11T21:19:18.673122+0000","last_became_active":"2024-03-11T21:18:52.258897+0000","last_became_peered":"2024-03-11T21:18:52.258897+0000","last_unstale":"2024-03-11T21:19:18.673122+0000","last_undegraded":"2024-03-11T21:19:18.673122+0000","last_fullsized":"2024-03-11T21:19:18.673122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:26.180192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673455+0000","last_change":"2024-03-11T21:18:54.270670+0000","last_active":"2024-03-11T21:19:18.673455+0000","last_peered":"2024-03-11T21:19:18.673455+0000","last_clean":"2024-03-11T21:19:18.673455+0000","last_became_active":"2024-03-11T21:18:54.269378+0000","last_became_peered":"2024-03-11T21:18:54.269378+0000","last_unstale":"2024-03-11T21:19:18.673455+0000","last_undegraded":"2024-03-11T21:19:18.673455+0000","last_fullsized":"2024-03-11T21:19:18.673455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:53.253873+0000","last_clean_scrub_stamp":"2024-03-11T21:18:53.253873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:24:09.119334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.676842+0000","last_change":"2024-03-11T21:18:48.245458+0000","last_active":"2024-03-11T21:19:18.676842+0000","last_peered":"2024-03-11T21:19:18.676842+0000","last_clean":"2024-03-11T21:19:18.676842+0000","last_became_active":"2024-03-11T21:18:48.243569+0000","last_became_peered":"2024-03-11T21:18:48.243569+0000","last_unstale":"2024-03-11T21:19:18.676842+0000","last_undegraded":"2024-03-11T21:19:18.676842+0000","last_fullsized":"2024-03-11T21:19:18.676842+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:47.230322+0000","last_clean_scrub_stamp":"2024-03-11T21:18:47.230322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:57.823070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.675859+0000","last_change":"2024-03-11T21:18:50.253928+0000","last_active":"2024-03-11T21:19:18.675859+0000","last_peered":"2024-03-11T21:19:18.675859+0000","last_clean":"2024-03-11T21:19:18.675859+0000","last_became_active":"2024-03-11T21:18:50.253429+0000","last_became_peered":"2024-03-11T21:18:50.253429+0000","last_unstale":"2024-03-11T21:19:18.675859+0000","last_undegraded":"2024-03-11T21:19:18.675859+0000","last_fullsized":"2024-03-11T21:19:18.675859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:49.236729+0000","last_clean_scrub_stamp":"2024-03-11T21:18:49.236729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:56:27.094231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-11T21:19:18.673869+0000","last_change":"2024-03-11T21:18:52.261308+0000","last_active":"2024-03-11T21:19:18.673869+0000","last_peered":"2024-03-11T21:19:18.673869+0000","last_clean":"2024-03-11T21:19:18.673869+0000","last_became_active":"2024-03-11T21:18:52.260499+0000","last_became_peered":"2024-03-11T21:18:52.260499+0000","last_unstale":"2024-03-11T21:19:18.673869+0000","last_undegraded":"2024-03-11T21:19:18.673869+0000","last_fullsized":"2024-03-11T21:19:18.673869+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:18:51.243891+0000","last_clean_scrub_stamp":"2024-03-11T21:18:51.243891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:32:59.436003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738392,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:19:37.496 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:19:37.496 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T21:19:37.496 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T21:19:37.496 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T21:19:37.674 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:37 smithi017 ceph-mon[29862]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:19:37.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:37 smithi001 ceph-mon[25774]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:19:37.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:37 smithi204 ceph-mon[30291]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:19:38.189 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:38.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:38 smithi001 ceph-mon[25774]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:38.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:38 smithi017 ceph-mon[29862]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:38.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:38 smithi204 ceph-mon[30291]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:38.955 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:38 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ::ffff:172.21.15.17 - - [11/Mar/2024:21:19:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:40.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-03-11T21:19:40.181 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T21:19:40.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:40 smithi017 ceph-mon[29862]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:40.675 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:40 smithi017 ceph-mon[29862]: from='client.? 172.21.15.1:0/3227716846' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:19:40.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:40 smithi001 ceph-mon[25774]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:40.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:40 smithi001 ceph-mon[25774]: from='client.? 172.21.15.1:0/3227716846' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:19:40.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:40 smithi204 ceph-mon[30291]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:40.802 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:40 smithi204 ceph-mon[30291]: from='client.? 172.21.15.1:0/3227716846' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:19:40.955 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:19:40 smithi001 systemd[1]: Started Ceph alertmanager.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:19:41.245 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:41] ENGINE Bus STOPPING 2024-03-11T21:19:41.245 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:19:41.246 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:41] ENGINE Bus STOPPED 2024-03-11T21:19:41.246 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:41] ENGINE Bus STARTING 2024-03-11T21:19:41.246 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:41] ENGINE Serving on http://:::9283 2024-03-11T21:19:41.246 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:19:41] ENGINE Bus STARTED 2024-03-11T21:19:41.271 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T21:19:41.271 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T21:19:41.271 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T21:19:41.281 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-03-11T21:19:41.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T21:19:41.522 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:41.906 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:41.906 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.906 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:41.907 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:41 smithi001 ceph-mon[25774]: Deploying daemon grafana.a on smithi017 2024-03-11T21:19:42.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:42.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:19:42.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:19:42.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:19:42.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.176 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:41 smithi017 ceph-mon[29862]: Deploying daemon grafana.a on smithi017 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:19:42.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:19:42.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:19:42.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:42.303 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:41 smithi204 ceph-mon[30291]: Deploying daemon grafana.a on smithi017 2024-03-11T21:19:42.956 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:19:42 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a[58359]: ts=2024-03-11T21:19:42.730Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000972535s 2024-03-11T21:19:43.195 INFO:teuthology.orchestra.run.smithi001.stdout:167 167 2024-03-11T21:19:43.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T21:19:44.098 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:44.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:43 smithi017 ceph-mon[29862]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:44.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:43 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:44.200 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:43 smithi001 ceph-mon[25774]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:44.200 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:43 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:44.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:43 smithi204 ceph-mon[30291]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:44.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:43 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:19:46.014 INFO:teuthology.orchestra.run.smithi001.stdout:Backend: cephadm 2024-03-11T21:19:46.014 INFO:teuthology.orchestra.run.smithi001.stdout:Available: Yes 2024-03-11T21:19:46.014 INFO:teuthology.orchestra.run.smithi001.stdout:Paused: No 2024-03-11T21:19:46.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:45 smithi017 ceph-mon[29862]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:46.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:45 smithi001 ceph-mon[25774]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:46.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:45 smithi204 ceph-mon[30291]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:46.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T21:19:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:47.133 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:46 smithi001 ceph-mon[25774]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:47.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:46 smithi017 ceph-mon[29862]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:47.203 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:46 smithi204 ceph-mon[30291]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:47.955 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:47 smithi001 ceph-mon[25774]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:48.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:47 smithi017 ceph-mon[29862]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:48.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:47 smithi204 ceph-mon[30291]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:48.633 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T21:19:48.633 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager.a smithi001 *:9093,9094 starting - - - - 2024-03-11T21:19:48.633 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.x smithi017 *:8443,8765 running (2m) 27s ago 2m 448M - 19.0.0-1979-g2b0101eb 4133f8bc41bb e616fb4a5632 2024-03-11T21:19:48.633 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.y smithi001 *:9283,8765 running (4m) 23s ago 4m 477M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 201f2a6640e5 2024-03-11T21:19:48.633 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.z smithi204 *:8443,8765 running (2m) 28s ago 2m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 54b95e8edd1a 2024-03-11T21:19:48.633 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a smithi001 running (4m) 23s ago 4m 49.6M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 5aa3a2da4a2c 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b smithi017 running (2m) 27s ago 2m 41.6M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 28dccf6c3add 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c smithi204 running (2m) 28s ago 2m 45.9M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb c22009e5b03a 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (2m) 23s ago 2m 50.2M 15.6G 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c8b5c07a3 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi017 running (105s) 27s ago 105s 51.9M 15.6G 19.0.0-1979-g2b0101eb 4133f8bc41bb ce810f2d36ec 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi204 running (80s) 28s ago 80s 50.6M 16.6G 19.0.0-1979-g2b0101eb 4133f8bc41bb 2b6627b8528f 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus.a smithi017 *:9095 running (38s) 27s ago 38s 25.3M - 2.43.0 a07b618ecd1d 2974da217827 2024-03-11T21:19:48.634 INFO:teuthology.orchestra.run.smithi001.stdout:rgw.foo.a smithi001 *:80 running (62s) 23s ago 62s 77.7M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 636f008e3631 2024-03-11T21:19:48.645 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:48 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ::ffff:172.21.15.17 - - [11/Mar/2024:21:19:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:48.955 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:48 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:49.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:48 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:49.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:48 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:49.439 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T21:19:49.684 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:50.061 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:49 smithi001 ceph-mon[25774]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:50.061 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:49 smithi001 ceph-mon[25774]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:50.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:49 smithi017 ceph-mon[29862]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:50.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:49 smithi017 ceph-mon[29862]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:50.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:49 smithi204 ceph-mon[30291]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:50.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:49 smithi204 ceph-mon[30291]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:50.807 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:19:50 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a[58359]: ts=2024-03-11T21:19:50.731Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00294181s 2024-03-11T21:19:51.541 INFO:teuthology.orchestra.run.smithi001.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T21:19:51.541 INFO:teuthology.orchestra.run.smithi001.stdout:alertmanager ?:9093,9094 0/1 - 60s smithi001=a;count:1 2024-03-11T21:19:51.541 INFO:teuthology.orchestra.run.smithi001.stdout:grafana ?:3000 0/1 - 57s smithi017=a;count:1 2024-03-11T21:19:51.541 INFO:teuthology.orchestra.run.smithi001.stdout:mgr 3/3 31s ago 2m smithi001=y;smithi017=x;smithi204=z;count:3 2024-03-11T21:19:51.541 INFO:teuthology.orchestra.run.smithi001.stdout:mon 3/3 31s ago 3m smithi001:172.21.15.1=a;smithi017:172.21.15.17=b;smithi204:172.21.15.204=c;count:3 2024-03-11T21:19:51.542 INFO:teuthology.orchestra.run.smithi001.stdout:node-exporter ?:9100 0/3 - 63s smithi001=a;smithi017=b;smithi204=c;count:3 2024-03-11T21:19:51.542 INFO:teuthology.orchestra.run.smithi001.stdout:osd 3 31s ago - 2024-03-11T21:19:51.542 INFO:teuthology.orchestra.run.smithi001.stdout:prometheus ?:9095 1/1 30s ago 65s smithi017=a;count:1 2024-03-11T21:19:51.542 INFO:teuthology.orchestra.run.smithi001.stdout:rgw.foo.a ?:80 1/1 26s ago 64s smithi001=foo.a;count:1 2024-03-11T21:19:52.121 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:51 smithi001 ceph-mon[25774]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:52.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T21:19:52.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:51 smithi017 ceph-mon[29862]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:52.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:51 smithi204 ceph-mon[30291]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:52.388 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:53.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:52 smithi017 ceph-mon[29862]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:53.205 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:52 smithi001 ceph-mon[25774]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:53.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:52 smithi204 ceph-mon[30291]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:54.055 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:53 smithi001 ceph-mon[25774]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:54.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:53 smithi017 ceph-mon[29862]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:54.297 INFO:teuthology.orchestra.run.smithi001.stdout:HOST ADDR LABELS STATUS 2024-03-11T21:19:54.298 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 172.21.15.1 2024-03-11T21:19:54.298 INFO:teuthology.orchestra.run.smithi001.stdout:smithi017 172.21.15.17 2024-03-11T21:19:54.298 INFO:teuthology.orchestra.run.smithi001.stdout:smithi204 172.21.15.204 2024-03-11T21:19:54.298 INFO:teuthology.orchestra.run.smithi001.stdout:3 hosts in cluster 2024-03-11T21:19:54.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:53 smithi204 ceph-mon[30291]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:54.955 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:54 smithi001 ceph-mon[25774]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:55.083 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T21:19:55.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:54 smithi017 ceph-mon[29862]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:55.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:54 smithi204 ceph-mon[30291]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:55.325 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:56.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:55 smithi017 ceph-mon[29862]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:56.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:55 smithi001 ceph-mon[25774]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:56.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:55 smithi204 ceph-mon[30291]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513000D480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme1n1 ssd Linux_43d80df6bfb4a8ce5548 89.4G Yes 2m ago 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme2n1 ssd Linux_dee3708d1c6d56c7b8c4 89.4G Yes 2m ago 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme3n1 ssd Linux_bb4c69966ceb20c7725a 89.4G Yes 2m ago 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme4n1 ssd Linux_ceb1424241b6263cc77a 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi017 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7473003Q480DGN 447G No 106s ago Has a FileSystem, LVM detected 2024-03-11T21:19:57.418 INFO:teuthology.orchestra.run.smithi001.stdout:smithi017 /dev/nvme1n1 ssd Linux_1954de3b0db8e35e0ad7 89.4G Yes 106s ago 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi017 /dev/nvme2n1 ssd Linux_11258fd07a1ec08c7132 89.4G Yes 106s ago 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi017 /dev/nvme3n1 ssd Linux_c93fa56772d41125d7dc 89.4G Yes 106s ago 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi017 /dev/nvme4n1 ssd Linux_23ea537f8325c8a8025b 89.4G No 106s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi204 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GD400BGN 372G No 81s ago Has a FileSystem, LVM detected 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi204 /dev/nvme1n1 ssd Linux_ffc4e8a5307b0245fa3d 89.4G Yes 81s ago 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi204 /dev/nvme2n1 ssd Linux_f25d6f4942282c96e820 89.4G Yes 81s ago 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi204 /dev/nvme3n1 ssd Linux_e22d56a2119ffbf430e3 89.4G Yes 81s ago 2024-03-11T21:19:57.419 INFO:teuthology.orchestra.run.smithi001.stdout:smithi204 /dev/nvme4n1 ssd Linux_1496ef3e468e76286588 89.4G No 81s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:19:58.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:57 smithi017 ceph-mon[29862]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:58.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:57 smithi017 ceph-mon[29862]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:58.205 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:57 smithi001 ceph-mon[25774]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:58.205 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:57 smithi001 ceph-mon[25774]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:58.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:57 smithi204 ceph-mon[30291]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:19:58.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:57 smithi204 ceph-mon[30291]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:19:58.446 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T21:19:58.691 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:19:58.705 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:19:58 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ::ffff:172.21.15.17 - - [11/Mar/2024:21:19:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:00.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:19:59 smithi017 ceph-mon[29862]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:00.205 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:19:59 smithi001 ceph-mon[25774]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:00.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:19:59 smithi204 ceph-mon[30291]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: alertmanager 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: alertmanager 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: count: 1 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi001=a 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:18:51.308867Z' 2024-03-11T21:20:00.739 INFO:teuthology.orchestra.run.smithi001.stdout: ports: 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: - 9093 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: - 9094 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: running: 0 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: size: 1 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:events: 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:- 2024-03-11T21:19:40.944602Z service:alertmanager [INFO] "service was created" 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: grafana 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: grafana 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: count: 1 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.740 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi017=a 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:spec: 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: anonymous_access: true 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: protocol: https 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:18:53.900454Z' 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: ports: 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: - 3000 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: running: 0 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout: size: 1 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: mgr 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: mgr 2024-03-11T21:20:00.741 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: count: 3 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi001=y 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi017=x 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi204=z 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:17:14.733473Z' 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: last_refresh: '2024-03-11T21:19:20.322701Z' 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: running: 3 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout: size: 3 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: mon 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: mon 2024-03-11T21:20:00.742 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: count: 3 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi001:172.21.15.1=a 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi017:172.21.15.17=b 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi204:172.21.15.204=c 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:16:49.022355Z' 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: last_refresh: '2024-03-11T21:19:20.322599Z' 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: running: 3 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout: size: 3 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: node-exporter 2024-03-11T21:20:00.743 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: node-exporter 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: count: 3 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi001=a 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi017=b 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi204=c 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:18:48.518312Z' 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: ports: 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: - 9100 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: running: 0 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout: size: 3 2024-03-11T21:20:00.744 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: osd 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: osd 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout:unmanaged: true 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout:spec: 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: filter_logic: AND 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: objectstore: bluestore 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: container_image_id: 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: last_refresh: '2024-03-11T21:19:20.322783Z' 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: running: 3 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout: size: 3 2024-03-11T21:20:00.745 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: prometheus 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: prometheus 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: count: 1 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi017=a 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:18:45.828056Z' 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: last_refresh: '2024-03-11T21:19:21.088303Z' 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: ports: 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: - 9095 2024-03-11T21:20:00.746 INFO:teuthology.orchestra.run.smithi001.stdout: running: 1 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: size: 1 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout:--- 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout:service_type: rgw 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout:service_id: foo.a 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout:service_name: rgw.foo.a 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout:placement: 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: count: 1 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: hosts: 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: - smithi001=foo.a 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout:status: 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: created: '2024-03-11T21:18:46.780475Z' 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: last_refresh: '2024-03-11T21:19:25.096587Z' 2024-03-11T21:20:00.747 INFO:teuthology.orchestra.run.smithi001.stdout: ports: 2024-03-11T21:20:00.748 INFO:teuthology.orchestra.run.smithi001.stdout: - 80 2024-03-11T21:20:00.748 INFO:teuthology.orchestra.run.smithi001.stdout: running: 1 2024-03-11T21:20:00.748 INFO:teuthology.orchestra.run.smithi001.stdout: size: 1 2024-03-11T21:20:01.062 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:00 smithi001 ceph-mon[25774]: overall HEALTH_OK 2024-03-11T21:20:01.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:00 smithi017 ceph-mon[29862]: overall HEALTH_OK 2024-03-11T21:20:01.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:00 smithi204 ceph-mon[30291]: overall HEALTH_OK 2024-03-11T21:20:01.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 415704e2-dfec-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T21:20:01.742 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:20:02.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:01 smithi017 ceph-mon[29862]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:02.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:01 smithi017 ceph-mon[29862]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:20:02.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:01 smithi001 ceph-mon[25774]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:02.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:01 smithi001 ceph-mon[25774]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:20:02.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:01 smithi204 ceph-mon[30291]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:02.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:01 smithi204 ceph-mon[30291]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:20:03.758 INFO:teuthology.orchestra.run.smithi001.stdout:osd 3 43s ago - 2024-03-11T21:20:04.138 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:03 smithi017 ceph-mon[29862]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:04.138 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:03 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:04.138 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:03 smithi017 podman[42113]: 2024-03-11 21:20:03.837886627 +0000 UTC m=+0.018526326 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T21:20:04.138 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 podman[42113]: 2024-03-11 21:20:04.153707512 +0000 UTC m=+0.334347212 container create 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a, name=ubi8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-type=git, version=8.5, io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, maintainer=Paul Cuzner ) 2024-03-11T21:20:04.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:03 smithi001 ceph-mon[25774]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:04.206 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:03 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:04.301 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:03 smithi204 ceph-mon[30291]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:20:04.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:03 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:04.406 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 podman[42113]: 2024-03-11 21:20:04.339114222 +0000 UTC m=+0.519753934 container init 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a, io.openshift.expose-services=, release=239.1651231664, version=8.5, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0) 2024-03-11T21:20:04.406 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 podman[42113]: 2024-03-11 21:20:04.344664462 +0000 UTC m=+0.525304162 container start 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, vendor=Red Hat, Inc., release=239.1651231664, io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5) 2024-03-11T21:20:04.529 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:20:04.546 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:20:04.547 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:04.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:04.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:04.645 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:20:04.645 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:20:04.656 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 bash[42113]: 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 systemd[1]: Started Ceph grafana.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.514885721Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515086119Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515120788Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515142119Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515153709Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515164998Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515176086Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515187631Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.51519799Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.51520747Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515224162Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515236955Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T21:20:04.657 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.51525184Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T21:20:04.658 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=settings t=2024-03-11T21:20:04.515263331Z level=info msg="App mode production" 2024-03-11T21:20:04.658 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=sqlstore t=2024-03-11T21:20:04.515336649Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T21:20:04.658 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=sqlstore t=2024-03-11T21:20:04.515363094Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T21:20:04.658 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.516551129Z level=info msg="Starting DB migrations" 2024-03-11T21:20:04.658 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.516648157Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.517234921Z level=info msg="Executing migration" id="create user table" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.517695849Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.518163549Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.518610015Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.519157185Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.519582999Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.520251222Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.520645801Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.521009577Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T21:20:04.659 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.521430781Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.521766677Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.522073082Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.52247975Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.522772886Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.523134434Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.52350584Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.52387218Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.524235083Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.524641444Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.525761634Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.526172297Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.526604592Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.527025426Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T21:20:04.660 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.527436544Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.527855168Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.528127208Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.528544374Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.528934422Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.52929745Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.52969862Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.530291877Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.530685462Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.531050802Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.531534513Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.531947603Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.532436952Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.532744456Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T21:20:04.661 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.533031581Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.533356956Z level=info msg="Executing migration" id="create star table" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.533691695Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.534072572Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.534447Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.534822144Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.535162011Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.53554145Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.535941214Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.536310571Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.53657604Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.536804911Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T21:20:04.662 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.537053703Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.53741064Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.537791175Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.538157109Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.538527025Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.538909449Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.539254008Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.540075251Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.540446036Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.540831031Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.541209929Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.541546856Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.541875094Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.542108282Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T21:20:04.663 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.542578431Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.543042252Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.543508584Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.543913025Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.544372819Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.544769586Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.545119633Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.545388579Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.545647315Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.546197282Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.546686503Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.547124885Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.54759959Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.547889788Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.548339067Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.548777353Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T21:20:04.664 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.549029656Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.549488207Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.549871087Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.550754277Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.551141852Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.5515483Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.55196401Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.552261687Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.552613198Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.553056089Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.55347202Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.55376696Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.554024189Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.55445851Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.555013561Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T21:20:04.665 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.555432321Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T21:20:04.666 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.555869448Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T21:20:04.667 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.556304256Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.556732371Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.557094707Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.558103009Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.55851086Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.558942079Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.559396799Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.559738635Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.560436868Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.560958883Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.561202061Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.561518549Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.562058177Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T21:20:04.669 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.56236492Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.56265653Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.563127683Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.56344717Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.563869429Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.564281779Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.564670989Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.565076002Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.565489035Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.56595645Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.566384151Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.566788142Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.567142798Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T21:20:04.670 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.568245194Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.568617405Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.568977802Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.569393595Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.569814295Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.570091792Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.570435654Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57069812Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.571180299Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57171739Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57198343Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57251283Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.573050985Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.573426776Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T21:20:04.671 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.573766861Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.574102645Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.574524794Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.574946867Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57535017Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.575627069Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.575864242Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.576399989Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.576933853Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.577185624Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.577599331Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.578021032Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.578276085Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T21:20:04.672 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57864591Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.579061755Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.579647567Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.57991886Z level=info msg="Executing migration" id="create session table" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.580323842Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.580614184Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.580864409Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.581226362Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.581590974Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.581829603Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.582066587Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.582369715Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.582652762Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.583014105Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T21:20:04.673 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.583260218Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.583834644Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.584090519Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.584676247Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.585251291Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.585533033Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.585955028Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.586443326Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.586877309Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.587312718Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.587734457Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.588150138Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.588520287Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T21:20:04.674 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.589258929Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.589890662Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.591632792Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.592001042Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.592436665Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.592739487Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.593044112Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.593417239Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.594043303Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.594665283Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.595289865Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.595899761Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.596314203Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.59658794Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T21:20:04.675 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.59681573Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.597178975Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.597593597Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.597977561Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.598362227Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.598751534Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.599360179Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.599980643Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.600244399Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.600653341Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.600993143Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.601646369Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.601915162Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.602281768Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T21:20:04.676 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.602667247Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.602916947Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.603335707Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.603754639Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.604145233Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.604562841Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.605003722Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.605501056Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.60577957Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.606442308Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.606843534Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.607503631Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.607898137Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.608377306Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T21:20:04.677 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.608772769Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.610349452Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.610724844Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.61114429Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.611463217Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.611800492Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.612065479Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.612767431Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.61343395Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.613917062Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.614330326Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.614644667Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.61527292Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.615706677Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T21:20:04.678 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.615970824Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.61629199Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.616700318Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.617070399Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.617546131Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.617952616Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.618352907Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.618759534Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.619027655Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.619432412Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.619805486Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.620239555Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.620655361Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.62092207Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.621197667Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T21:20:04.679 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.621496004Z level=info msg="Executing migration" id="create team table" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.621886389Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.622372144Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.622789839Z level=info msg="Executing migration" id="create team member table" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.623159352Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.623591193Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.624007401Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.624479164Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.625162816Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.625859573Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.626554337Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.626945712Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.627400026Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.627814506Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T21:20:04.680 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.62820971Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.628741186Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.629130237Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.629561408Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.629954171Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.630302481Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.630600136Z level=info msg="Executing migration" id="create tag table" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.630946006Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.631352343Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.631717917Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.632113814Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.632505702Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.634390478Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.634753195Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T21:20:04.681 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.635146054Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.635446262Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.635780402Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.636116142Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.636526683Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.636799298Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.637515741Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.638272413Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.639005498Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.639736349Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.6401383Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.640859583Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.641239237Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T21:20:04.682 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.641661676Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.642065498Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.642493976Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.642891463Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.643296405Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.64403801Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.644454821Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.644860662Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.64526461Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.645669928Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.645906307Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.64615672Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.646627005Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.647022072Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T21:20:04.683 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.647426113Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.647690755Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.648042486Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.64843973Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.648847114Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.649248113Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.650020286Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.650424324Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.650707473Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.651047136Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.651459695Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.651854651Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.652085279Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T21:20:04.684 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.652476022Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.65288178Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.65325343Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.653654322Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.654437637Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.654824449Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.655174826Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.658633619Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.662017487Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.66244466Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.662842785Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.663672766Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.664059376Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.664474999Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T21:20:04.685 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.66487912Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.665285972Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.665559504Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.666345646Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.667122654Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.667898851Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.668306278Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.668712375Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.669520547Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.67030039Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.670729824Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.671549955Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.672365171Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T21:20:04.686 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.672659884Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.673112614Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.673556779Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.673994026Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.674251083Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.675136211Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.676022205Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.676884409Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.677745796Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.678595948Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.678894412Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.679289543Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.680130426Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.680420126Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.681307592Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T21:20:04.687 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.681742063Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T21:20:04.688 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.682614025Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T21:20:04.701 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:20:04.718 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:20:04.721 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:20:04.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.683007502Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.683443122Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.684323634Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.684760175Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.685187947Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.685599153Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T21:20:04.918 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.686029235Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.686288763Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.686692596Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.687088465Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.687173708Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.687428576Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.687823014Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.688214322Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.688348009Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.688490757Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.688931743Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.68939018Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.689872966Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T21:20:04.919 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.690308771Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.690804498Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.691326951Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.691625968Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.691999588Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.692465765Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.692926979Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.698671944Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.699579239Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.699856546Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.704141506Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.708451665Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.708837929Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.709255235Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T21:20:04.920 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.709572665Z level=info msg="Executing migration" id="create permission table" 2024-03-11T21:20:04.921 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.710010446Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T21:20:04.921 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.710426354Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T21:20:04.921 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.710837875Z level=info msg="Executing migration" id="create role table" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.711198426Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.712095617Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.713034144Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.713451028Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.71385604Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.714266232Z level=info msg="Executing migration" id="create team role table" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.714625629Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T21:20:04.922 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.715029961Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.715470036Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.715873546Z level=info msg="Executing migration" id="create user role table" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.716245153Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.716647125Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.717046244Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.717455652Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.717844104Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.718255282Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.71865695Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.7196054Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.720047783Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.720463819Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T21:20:04.923 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.720852977Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.721216767Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.7216049Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.722006145Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.722939987Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.723333528Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.723747963Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.723986638Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.724428136Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.724875059Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.725182532Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.725471058Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.725784973Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.726154227Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.726581368Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.727538144Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T21:20:04.924 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.727811745Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.72819778Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.728614303Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.729023922Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.729982682Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.730423177Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.730809673Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.730889683Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.731141204Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.731225329Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.731477912Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.731837355Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.73219462Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:20:04.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.732620161Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.733100661Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.73349763Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.733888694Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.734241785Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.734637794Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.735015823Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.735451405Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.735872687Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.738876258Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.73986041Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.740841506Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.741128132Z level=info msg="Executing migration" id="add share column" 2024-03-11T21:20:04.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.742100584Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.742514238Z level=info msg="Executing migration" id="create file table" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.742911052Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.743330753Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.743775074Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.744125551Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.7445347Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.744792703Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.745095094Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.745386858Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.745797547Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.746784061Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.747063357Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.747485649Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.747852828Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T21:20:04.927 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.748113688Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.748457323Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.749448686Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.750441434Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.750852036Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.760887946Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.761308071Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.761746168Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.765181797Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.765489171Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.765777688Z level=info msg="Executing migration" id="create folder table" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.766122565Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.766537002Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.766947721Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.767171979Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T21:20:04.928 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=migrator t=2024-03-11T21:20:04.767584051Z level=info msg="migrations completed" performed=485 skipped=0 duration=250.969475ms 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=sqlstore t=2024-03-11T21:20:04.767980272Z level=info msg="Created default organization" 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=plugin.loader t=2024-03-11T21:20:04.825596126Z level=info msg="Plugin registered" pluginID=input 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=plugin.loader t=2024-03-11T21:20:04.854873516Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=plugin.loader t=2024-03-11T21:20:04.854893405Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=secrets t=2024-03-11T21:20:04.854929261Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=query_data t=2024-03-11T21:20:04.857117005Z level=info msg="Query Service initialization" 2024-03-11T21:20:04.929 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=live.push_http t=2024-03-11T21:20:04.861178585Z level=info msg="Live Push Gateway initialization" 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:04 smithi001 systemd[1]: Stopping Ceph mon.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:04 smithi001 ceph-mon[25774]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:04 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:04 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:04 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:04 smithi001 ceph-mon[25774]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a[25751]: 2024-03-11T21:20:05.070+0000 7fca8ec76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:05.057 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a[25751]: 2024-03-11T21:20:05.070+0000 7fca8ec76700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:20:05.174 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-mon[29862]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.175 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:04 smithi017 ceph-mon[29862]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:04 smithi204 ceph-mon[30291]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:04 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:04 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:04 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.302 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:04 smithi204 ceph-mon[30291]: from='mgr.14439 172.21.15.1:0/594324963' entity='mgr.y' 2024-03-11T21:20:05.382 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:05 smithi001 podman[60229]: 2024-03-11 21:20:05.091981151 +0000 UTC m=+0.153452050 container died 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T21:20:05.382 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:05] ENGINE Bus STOPPING 2024-03-11T21:20:05.382 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:20:05.382 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:05] ENGINE Bus STOPPED 2024-03-11T21:20:05.382 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:05] ENGINE Bus STARTING 2024-03-11T21:20:05.383 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:05] ENGINE Serving on http://:::9283 2024-03-11T21:20:05.383 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:05 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:05] ENGINE Bus STARTED 2024-03-11T21:20:05.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:05 smithi001 podman[60229]: 2024-03-11 21:20:05.613847005 +0000 UTC m=+0.675317946 container cleanup 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T21:20:05.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:05 smithi001 bash[60229]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:20:05.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=infra.usagestats.collector t=2024-03-11T21:20:05.508761207Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T21:20:05.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=provisioning.datasources t=2024-03-11T21:20:05.509715351Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T21:20:05.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=provisioning.datasources t=2024-03-11T21:20:05.521023408Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T21:20:05.925 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=provisioning.alerting t=2024-03-11T21:20:05.532231742Z level=info msg="starting to provision alerting" 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=provisioning.alerting t=2024-03-11T21:20:05.532252076Z level=info msg="finished to provision alerting" 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=grafanaStorageLogger t=2024-03-11T21:20:05.532445007Z level=info msg="storage starting" 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=ngalert.state.manager t=2024-03-11T21:20:05.532505317Z level=info msg="Warming state cache for startup" 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=http.server t=2024-03-11T21:20:05.534725Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=ngalert.state.manager t=2024-03-11T21:20:05.538351807Z level=info msg="State cache has been initialized" states=0 duration=5.845747ms 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=ticker t=2024-03-11T21:20:05.538410586Z level=info msg=starting first_tick=2024-03-11T21:20:10Z 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=ngalert.multiorg.alertmanager t=2024-03-11T21:20:05.538427398Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T21:20:05.926 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:05 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a[42127]: logger=sqlstore.transactions t=2024-03-11T21:20:05.554228015Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:20:06.159 INFO:journalctl@ceph.mon.a.smithi001.stdout:Mar 11 21:20:05 smithi001 podman[60262]: 2024-03-11 21:20:05.815062548 +0000 UTC m=+0.722502579 container remove 5aa3a2da4a2c3c1568c9543ac116ce62546b7b43007060d7a73c0e8d77152b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:20:06.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:20:06.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:06.378 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:20:06.378 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T21:20:06.378 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:20:06.674 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:06 smithi017 systemd[1]: Stopping Ceph mon.b for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:06.934 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:06 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b[29839]: 2024-03-11T21:20:06.693+0000 7f667583f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:06.935 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:06 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b[29839]: 2024-03-11T21:20:06.694+0000 7f667583f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:20:06.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:06] ENGINE Bus STOPPING 2024-03-11T21:20:06.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:20:06.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:06] ENGINE Bus STOPPED 2024-03-11T21:20:06.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:06] ENGINE Bus STARTING 2024-03-11T21:20:06.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:06] ENGINE Serving on http://:::9283 2024-03-11T21:20:06.956 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:06 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: [11/Mar/2024:21:20:06] ENGINE Bus STARTED 2024-03-11T21:20:07.185 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:06 smithi017 podman[42247]: 2024-03-11 21:20:06.950682907 +0000 UTC m=+0.359360164 container died 28dccf6c3addd7b0cb8410f8aadb19e8701dab8e334849551877b3f0ed3e8403 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T21:20:07.455 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:07 smithi001 bash[60489]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T21:20:07.468 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:07 smithi017 podman[42247]: 2024-03-11 21:20:07.300250143 +0000 UTC m=+0.708927403 container cleanup 28dccf6c3addd7b0cb8410f8aadb19e8701dab8e334849551877b3f0ed3e8403 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T21:20:07.468 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:07 smithi017 bash[42247]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:20:07.718 INFO:journalctl@ceph.mon.b.smithi017.stdout:Mar 11 21:20:07 smithi017 podman[42267]: 2024-03-11 21:20:07.484399718 +0000 UTC m=+0.534926012 container remove 28dccf6c3addd7b0cb8410f8aadb19e8701dab8e334849551877b3f0ed3e8403 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T21:20:07.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:20:07.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:07.770 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T21:20:07.770 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T21:20:07.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:20:08.052 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:07 smithi204 systemd[1]: Stopping Ceph mon.c for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:08.475 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:08 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c[30268]: 2024-03-11T21:20:08.085+0000 7f6ed4d08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:08.475 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:08 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c[30268]: 2024-03-11T21:20:08.085+0000 7f6ed4d08700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:20:08.475 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:08 smithi204 podman[43320]: 2024-03-11 21:20:08.126845374 +0000 UTC m=+0.135270561 container died c22009e5b03a0de2b12fb288c83c9c15766e04978cd6479718d44bc2dd172923 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:20:08.475 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:08 smithi204 podman[43320]: 2024-03-11 21:20:08.491034 +0000 UTC m=+0.499459186 container cleanup c22009e5b03a0de2b12fb288c83c9c15766e04978cd6479718d44bc2dd172923 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T21:20:08.759 INFO:journalctl@ceph.mon.c.smithi204.stdout:Mar 11 21:20:08 smithi204 bash[43320]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:20:08.889 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:20:08.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:08.934 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T21:20:08.934 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:20:08.935 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:20:08.955 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:08 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: ::ffff:172.21.15.17 - - [11/Mar/2024:21:20:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:09.251 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:09 smithi001 systemd[1]: Stopping Ceph mgr.y for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:09.599 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:09 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:20:09.265+0000 7ff4af937700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:09.600 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:09 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y[26035]: 2024-03-11T21:20:09.265+0000 7ff4af937700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T21:20:09.600 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:09 smithi001 podman[60582]: 2024-03-11 21:20:09.297774692 +0000 UTC m=+0.157273486 container died 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T21:20:09.935 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:09 smithi001 podman[60582]: 2024-03-11 21:20:09.865390732 +0000 UTC m=+0.724889519 container cleanup 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T21:20:09.935 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:09 smithi001 bash[60582]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y 2024-03-11T21:20:10.345 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 bash[60489]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T21:20:10.345 INFO:journalctl@ceph.mgr.y.smithi001.stdout:Mar 11 21:20:10 smithi001 podman[60602]: 2024-03-11 21:20:10.268956512 +0000 UTC m=+0.971862880 container remove 201f2a6640e5eaa3007c32caf9ad9548e0db292436eb4fe4cc424f520bbd5b98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-11T21:20:10.675 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:10.278Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.1:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.1:8765: connect: connection refused" 2024-03-11T21:20:10.675 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:10.278Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.1:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.1:8765: connect: connection refused" 2024-03-11T21:20:10.675 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:10 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:10.280Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.1:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.1:8765: connect: connection refused" 2024-03-11T21:20:10.689 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 bash[60489]: Writing manifest to image destination 2024-03-11T21:20:10.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:20:10.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:10.856 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:20:10.856 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:20:10.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 podman[60489]: 2024-03-11 21:20:10.705830184 +0000 UTC m=+4.220917257 container create 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 podman[60489]: 2024-03-11 21:20:10.615877751 +0000 UTC m=+4.130964837 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 podman[60489]: 2024-03-11 21:20:10.916594968 +0000 UTC m=+4.431682058 container init 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 podman[60489]: 2024-03-11 21:20:10.9223853 +0000 UTC m=+4.437472374 container start 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.926Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.926Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:20:10.956 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.926Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.926Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:20:10.957 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:20:10.958 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:20:10.959 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:20:10.960 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:20:10.961 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:20:10.961 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:20:10.961 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:10 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a[60663]: ts=2024-03-11T21:20:10.927Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:20:11.174 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:20:10 smithi017 systemd[1]: Stopping Ceph mgr.x for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:11.457 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:11 smithi001 bash[60489]: 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 2024-03-11T21:20:11.457 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:11 smithi001 systemd[1]: Started Ceph node-exporter.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:20:11.463 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:20:11 smithi017 podman[42380]: 2024-03-11 21:20:11.194287184 +0000 UTC m=+0.131024853 container died e616fb4a563295fdcb99bfa7f3c8b67285449e67b4123a95c41303d24bef8656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T21:20:11.772 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:20:11 smithi017 podman[42380]: 2024-03-11 21:20:11.587135013 +0000 UTC m=+0.523872682 container cleanup e616fb4a563295fdcb99bfa7f3c8b67285449e67b4123a95c41303d24bef8656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T21:20:11.772 INFO:journalctl@ceph.mgr.x.smithi017.stdout:Mar 11 21:20:11 smithi017 bash[42380]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-x 2024-03-11T21:20:11.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:20:11.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:11.934 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:20:11.934 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T21:20:11.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.z 2024-03-11T21:20:12.261 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:20:12 smithi204 systemd[1]: Stopping Ceph mgr.z for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:12.552 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:20:12 smithi204 podman[43451]: 2024-03-11 21:20:12.27749989 +0000 UTC m=+0.133679312 container died 54b95e8edd1a4de9b3634a16fb297498df4ee78b087ae9cb2bf92be132b653e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:20:12.897 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:20:12 smithi204 podman[43451]: 2024-03-11 21:20:12.66160324 +0000 UTC m=+0.517782660 container cleanup 54b95e8edd1a4de9b3634a16fb297498df4ee78b087ae9cb2bf92be132b653e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T21:20:12.897 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Mar 11 21:20:12 smithi204 bash[43451]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-mgr-z 2024-03-11T21:20:13.008 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T21:20:13.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:13.043 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T21:20:13.043 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:20:13.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:20:13.339 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:13 smithi001 systemd[1]: Stopping Ceph osd.0 for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:13.705 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0[44215]: 2024-03-11T21:20:13.353+0000 7f0ffb1d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:13.706 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0[44215]: 2024-03-11T21:20:13.353+0000 7f0ffb1d5700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-11T21:20:13.706 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:13 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0[44215]: 2024-03-11T21:20:13.353+0000 7f0ffb1d5700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:20:18.956 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:18 smithi001 podman[60758]: 2024-03-11 21:20:18.68574016 +0000 UTC m=+5.424464121 container died bc9c8b5c07a329b80302d1e9ebbabc6c0cf1ed45997c6cd15d1ea5de8bef8b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-11T21:20:19.414 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 podman[60758]: 2024-03-11 21:20:19.187118281 +0000 UTC m=+5.925842283 container cleanup bc9c8b5c07a329b80302d1e9ebbabc6c0cf1ed45997c6cd15d1ea5de8bef8b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-11T21:20:19.414 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 bash[60758]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:20:19.705 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 podman[60781]: 2024-03-11 21:20:19.430027063 +0000 UTC m=+0.740137708 container remove bc9c8b5c07a329b80302d1e9ebbabc6c0cf1ed45997c6cd15d1ea5de8bef8b93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:20:20.051 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 podman[60878]: 2024-03-11 21:20:19.653053071 +0000 UTC m=+0.018923817 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:20:20.051 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 podman[60878]: 2024-03-11 21:20:19.755697809 +0000 UTC m=+0.121568541 container create 7394e0ba535865461109051664b59ae55f1017a8841714e4e533062d073530d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-11T21:20:20.051 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 podman[60878]: 2024-03-11 21:20:19.983603633 +0000 UTC m=+0.349474365 container init 7394e0ba535865461109051664b59ae55f1017a8841714e4e533062d073530d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:20:20.051 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:19 smithi001 podman[60878]: 2024-03-11 21:20:19.988940101 +0000 UTC m=+0.354810833 container start 7394e0ba535865461109051664b59ae55f1017a8841714e4e533062d073530d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-11T21:20:20.051 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:20 smithi001 podman[60878]: 2024-03-11 21:20:20.066529574 +0000 UTC m=+0.432400324 container attach 7394e0ba535865461109051664b59ae55f1017a8841714e4e533062d073530d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T21:20:20.456 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:20 smithi001 podman[60878]: 2024-03-11 21:20:20.283816313 +0000 UTC m=+0.649687045 container died 7394e0ba535865461109051664b59ae55f1017a8841714e4e533062d073530d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T21:20:20.872 INFO:journalctl@ceph.osd.0.smithi001.stdout:Mar 11 21:20:20 smithi001 podman[60925]: 2024-03-11 21:20:20.886180293 +0000 UTC m=+0.597762705 container remove 7394e0ba535865461109051664b59ae55f1017a8841714e4e533062d073530d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T21:20:20.887 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:20:20.930 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:20.931 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:20:20.931 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:20:20.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:20:21.237 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:20 smithi017 systemd[1]: Stopping Ceph osd.1 for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:21.674 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:21 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1[34424]: 2024-03-11T21:20:21.251+0000 7f76078ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:21.675 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:21 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1[34424]: 2024-03-11T21:20:21.251+0000 7f76078ef700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-11T21:20:21.675 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:21 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1[34424]: 2024-03-11T21:20:21.251+0000 7f76078ef700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:20:26.800 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:26 smithi017 podman[42513]: 2024-03-11 21:20:26.563323284 +0000 UTC m=+5.418368730 container died ce810f2d36ec652a6c322120d601e83644959a5027aea63a83d8b6f33a25666d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:20:27.075 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:26 smithi017 podman[42513]: 2024-03-11 21:20:26.915351176 +0000 UTC m=+5.770396636 container cleanup ce810f2d36ec652a6c322120d601e83644959a5027aea63a83d8b6f33a25666d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:20:27.076 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:26 smithi017 bash[42513]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:20:27.424 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:27 smithi017 podman[42546]: 2024-03-11 21:20:27.091542042 +0000 UTC m=+0.527267374 container remove ce810f2d36ec652a6c322120d601e83644959a5027aea63a83d8b6f33a25666d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-11T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:27 smithi017 podman[42644]: 2024-03-11 21:20:27.500631607 +0000 UTC m=+0.114291794 container create d5d222d162c310b6626f75a85d6922e1089188de13acad798dcf7c4ae070560b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:27 smithi017 podman[42644]: 2024-03-11 21:20:27.404906203 +0000 UTC m=+0.018566405 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:27 smithi017 podman[42644]: 2024-03-11 21:20:27.694448929 +0000 UTC m=+0.308109120 container init d5d222d162c310b6626f75a85d6922e1089188de13acad798dcf7c4ae070560b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:27 smithi017 podman[42644]: 2024-03-11 21:20:27.699570329 +0000 UTC m=+0.313230520 container start d5d222d162c310b6626f75a85d6922e1089188de13acad798dcf7c4ae070560b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T21:20:27.753 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:27 smithi017 podman[42644]: 2024-03-11 21:20:27.769210979 +0000 UTC m=+0.382871180 container attach d5d222d162c310b6626f75a85d6922e1089188de13acad798dcf7c4ae070560b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T21:20:28.174 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 11 21:20:28 smithi017 podman[42644]: 2024-03-11 21:20:28.001169008 +0000 UTC m=+0.614829209 container died d5d222d162c310b6626f75a85d6922e1089188de13acad798dcf7c4ae070560b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T21:20:28.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:20:28.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:28.576 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:20:28.576 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:20:28.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:20:28.877 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:28 smithi204 systemd[1]: Stopping Ceph osd.2 for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:29.301 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:28 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2[35570]: 2024-03-11T21:20:28.892+0000 7fd653a82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:29.301 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:28 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2[35570]: 2024-03-11T21:20:28.892+0000 7fd653a82700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-11T21:20:29.301 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:28 smithi204 ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2[35570]: 2024-03-11T21:20:28.892+0000 7fd653a82700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:20:34.482 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:34 smithi204 podman[43596]: 2024-03-11 21:20:34.208267298 +0000 UTC m=+5.421207957 container died 2b6627b8528f49d4c2ac2e5947ac22598f2b04bcf9c0e7450b4a49cf97dfae8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T21:20:34.782 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:34 smithi204 podman[43596]: 2024-03-11 21:20:34.597516043 +0000 UTC m=+5.810456699 container cleanup 2b6627b8528f49d4c2ac2e5947ac22598f2b04bcf9c0e7450b4a49cf97dfae8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T21:20:34.783 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:34 smithi204 bash[43596]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:20:35.051 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:34 smithi204 podman[43618]: 2024-03-11 21:20:34.798373033 +0000 UTC m=+0.580304235 container remove 2b6627b8528f49d4c2ac2e5947ac22598f2b04bcf9c0e7450b4a49cf97dfae8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e) 2024-03-11T21:20:35.410 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:35 smithi204 podman[43715]: 2024-03-11 21:20:35.033337327 +0000 UTC m=+0.017479118 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:20:35.410 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:35 smithi204 podman[43715]: 2024-03-11 21:20:35.14082601 +0000 UTC m=+0.124967847 container create f910bc503c5f28784384aff4c36764c079507aba5a867e4898d9686911c425eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:20:35.410 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:35 smithi204 podman[43715]: 2024-03-11 21:20:35.351361681 +0000 UTC m=+0.335503481 container init f910bc503c5f28784384aff4c36764c079507aba5a867e4898d9686911c425eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:20:35.410 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:35 smithi204 podman[43715]: 2024-03-11 21:20:35.356699409 +0000 UTC m=+0.340841200 container start f910bc503c5f28784384aff4c36764c079507aba5a867e4898d9686911c425eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:20:35.410 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:35 smithi204 podman[43715]: 2024-03-11 21:20:35.426135281 +0000 UTC m=+0.410277100 container attach f910bc503c5f28784384aff4c36764c079507aba5a867e4898d9686911c425eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0) 2024-03-11T21:20:35.802 INFO:journalctl@ceph.osd.2.smithi204.stdout:Mar 11 21:20:35 smithi204 podman[43715]: 2024-03-11 21:20:35.647576675 +0000 UTC m=+0.631718469 container died f910bc503c5f28784384aff4c36764c079507aba5a867e4898d9686911c425eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T21:20:36.254 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:20:36.293 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:36.293 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:20:36.293 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:20:36.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:20:36.599 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:36 smithi001 systemd[1]: Stopping Ceph rgw.foo.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:36.955 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:36 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a[50950]: 2024-03-11T21:20:36.614+0000 7f6e8afc0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:36.956 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:36 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a[50950]: 2024-03-11T21:20:36.614+0000 7f6e99838ac0 -1 shutting down 2024-03-11T21:20:46.906 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:46 smithi001 bash[61011]: time="2024-03-11T21:20:46Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:20:46.906 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:46 smithi001 podman[61011]: 2024-03-11 21:20:46.643618126 +0000 UTC m=+10.139348793 container died 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T21:20:47.205 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:47 smithi001 podman[61011]: 2024-03-11 21:20:47.086618847 +0000 UTC m=+10.582349509 container cleanup 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-11T21:20:47.206 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:47 smithi001 bash[61011]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:20:47.507 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:47 smithi001 podman[61048]: 2024-03-11 21:20:47.279853451 +0000 UTC m=+0.627767476 container remove 636f008e363178b0b17726f01356cebd81b7682bcaa2ba6315d07bb83b86491c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T21:20:47.508 INFO:journalctl@ceph.rgw.foo.a.smithi001.stdout:Mar 11 21:20:47 smithi001 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:20:47.512 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:20:47.551 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:47.551 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:20:47.551 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:20:47.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-415704e2-dfec-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:20:47.857 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 systemd[1]: Stopping Ceph prometheus.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:20:48.175 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:20:48.176 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:20:48.176 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:20:48.176 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a[40235]: ts=2024-03-11T21:20:47.873Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:20:48.176 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:47 smithi017 podman[42774]: 2024-03-11 21:20:47.883891754 +0000 UTC m=+0.120533769 container died 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:48.589 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:48 smithi017 podman[42774]: 2024-03-11 21:20:48.195002373 +0000 UTC m=+0.431644379 container cleanup 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:48.589 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:48 smithi017 bash[42774]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:20:48.589 INFO:journalctl@ceph.prometheus.a.smithi017.stdout:Mar 11 21:20:48 smithi017 podman[42793]: 2024-03-11 21:20:48.379572258 +0000 UTC m=+0.499594868 container remove 2974da2178277a447dd6419e839ca7020b7a3ce16b744715a55461db39999a0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:48.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-415704e2-dfec-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:20:48.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:20:48.644 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:20:48.644 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:20:48.839 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting cluster with fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:20:49.936 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:49 smithi001 systemd[1]: Stopping Ceph alertmanager.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:50.205 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:49 smithi001 ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a[58359]: ts=2024-03-11T21:20:49.952Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:20:50.206 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:49 smithi001 podman[61366]: 2024-03-11 21:20:49.983223746 +0000 UTC m=+0.140217611 container died 00f8c7242fb6edc4f91ba63013719749c5b069a1a7999b54afc3e0d035d85784 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:50.584 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:50 smithi001 podman[61366]: 2024-03-11 21:20:50.349433199 +0000 UTC m=+0.506427064 container cleanup 00f8c7242fb6edc4f91ba63013719749c5b069a1a7999b54afc3e0d035d85784 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:50.584 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:50 smithi001 bash[61366]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:20:50.955 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:50 smithi001 podman[61385]: 2024-03-11 21:20:50.600363579 +0000 UTC m=+0.616767761 container remove 00f8c7242fb6edc4f91ba63013719749c5b069a1a7999b54afc3e0d035d85784 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:50.955 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:50 smithi001 podman[61385]: 2024-03-11 21:20:50.675988664 +0000 UTC m=+0.692392859 volume remove e07f506fe61450ac2fcf60d6efb0354aecd790931496318f3c8af94f5f9aad3c 2024-03-11T21:20:50.955 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:50 smithi001 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:20:50.955 INFO:journalctl@ceph.alertmanager.a.smithi001.stdout:Mar 11 21:20:50 smithi001 systemd[1]: Stopped Ceph alertmanager.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:20:51.455 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 systemd[1]: Stopping Ceph node-exporter.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:51.455 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 podman[61503]: 2024-03-11 21:20:51.250430016 +0000 UTC m=+0.092349778 container died 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:51.755 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 podman[61503]: 2024-03-11 21:20:51.553263101 +0000 UTC m=+0.395182860 container cleanup 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:51.755 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 bash[61503]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:20:52.027 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 podman[61522]: 2024-03-11 21:20:51.771085785 +0000 UTC m=+0.522165062 container remove 22de3a325d0c9338de4e13232d73e1366993e183788c1652e35ae6c8dcf0b764 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:20:52.027 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:20:52.027 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T21:20:52.027 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 systemd[1]: Stopped Ceph node-exporter.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:20:52.028 INFO:journalctl@ceph.node-exporter.a.smithi001.stdout:Mar 11 21:20:51 smithi001 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.857s CPU time. 2024-03-11T21:20:52.550 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:20:52.746 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:20:53.914 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:53 smithi017 systemd[1]: Stopping Ceph grafana.a for 415704e2-dfec-11ee-95c8-87774f69a715... 2024-03-11T21:20:54.174 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:53 smithi017 podman[43125]: 2024-03-11 21:20:53.929240329 +0000 UTC m=+0.119877267 container died 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, name=ubi8, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, architecture=x86_64) 2024-03-11T21:20:54.475 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 podman[43125]: 2024-03-11 21:20:54.298919253 +0000 UTC m=+0.489556189 container cleanup 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-04-29T11:48:11.055897, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vendor=Red Hat, Inc., description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.buildah.version=1.32.0) 2024-03-11T21:20:54.475 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 bash[43125]: ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:20:54.761 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 podman[43144]: 2024-03-11 21:20:54.491547671 +0000 UTC m=+0.565104427 container remove 2067a39f37444d52a47cad2e06391c44777f0f504d9b535d26886d4d90f8be02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-415704e2-dfec-11ee-95c8-87774f69a715-grafana-a, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vcs-type=git, vendor=Red Hat, Inc., version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.buildah.version=1.32.0, io.openshift.expose-services=, name=ubi8, build-date=2022-04-29T11:48:11.055897) 2024-03-11T21:20:54.761 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:20:54.761 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:20:54.761 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 systemd[1]: Stopped Ceph grafana.a for 415704e2-dfec-11ee-95c8-87774f69a715. 2024-03-11T21:20:54.761 INFO:journalctl@ceph.grafana.a.smithi017.stdout:Mar 11 21:20:54 smithi017 systemd[1]: ceph-415704e2-dfec-11ee-95c8-87774f69a715@grafana.a.service: Consumed 2.712s CPU time. 2024-03-11T21:20:55.249 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:20:55.442 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:20:56.486 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:56.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:56.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:56.584 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:20:56.586 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi001/crash 2024-03-11T21:20:56.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:20:56.624 INFO:teuthology.orchestra.run.smithi001.stderr:tar: /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:20:56.625 INFO:teuthology.orchestra.run.smithi001.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:20:56.626 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi017/crash 2024-03-11T21:20:56.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:20:56.671 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:20:56.671 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:20:56.673 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi204/crash 2024-03-11T21:20:56.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:20:56.715 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/415704e2-dfec-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:20:56.715 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:20:56.716 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:20:56.716 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:20:56.748 INFO:teuthology.orchestra.run.smithi001.stdout:2024-03-11T21:17:08.454036+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:20:56.748 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T21:20:56.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[SEC\]' /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:20:56.820 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]' /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:20:56.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[WRN\]' /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:20:56.963 INFO:teuthology.orchestra.run.smithi001.stdout:2024-03-11T21:17:08.454036+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:20:56.963 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:20:56.963 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:20:57.006 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:20:57.008 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:20:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:20:57.032 INFO:teuthology.orchestra.run.smithi001.stderr:: No such file or directory 2024-03-11T21:20:57.033 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr: 57.7% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.y.log: /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log 89.3% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:20:57.036 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-11T21:20:57.037 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:20:57.037 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log 82.0% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:20:57.037 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-11T21:20:57.037 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log: 84.8% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:20:57.041 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T21:20:57.041 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:20:57.041 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:20:57.041 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:20:57.041 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr: 89.4% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr: 80.2% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log: 81.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:20:57.042 INFO:teuthology.orchestra.run.smithi017.stderr: 84.2% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:20:57.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log: 90.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr: 80.1% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log: 89.5% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.2.log: /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log: 84.4% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T21:20:57.044 INFO:teuthology.orchestra.run.smithi204.stderr: 90.5% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:20:57.045 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log: 90.5% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:20:57.045 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.b.log: 90.5% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:20:57.045 INFO:teuthology.orchestra.run.smithi017.stderr: 90.3% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:20:57.092 INFO:teuthology.orchestra.run.smithi204.stderr: 92.2% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:20:57.092 INFO:teuthology.orchestra.run.smithi017.stderr: 92.3% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:20:57.103 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:20:57.208 INFO:teuthology.orchestra.run.smithi001.stderr: 91.4% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:20:57.291 INFO:teuthology.orchestra.run.smithi204.stderr: 94.3% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:20:57.293 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-11T21:20:57.293 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m0.270s 2024-03-11T21:20:57.293 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m0.316s 2024-03-11T21:20:57.293 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.033s 2024-03-11T21:20:57.298 INFO:teuthology.orchestra.run.smithi001.stderr: 94.2% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T21:20:57.300 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-03-11T21:20:57.300 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m0.281s 2024-03-11T21:20:57.301 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m0.513s 2024-03-11T21:20:57.301 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.040s 2024-03-11T21:20:57.354 INFO:teuthology.orchestra.run.smithi017.stderr: 94.3% -- replaced with /var/log/ceph/415704e2-dfec-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T21:20:57.355 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-11T21:20:57.355 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m0.333s 2024-03-11T21:20:57.356 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m0.378s 2024-03-11T21:20:57.356 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.040s 2024-03-11T21:20:57.356 INFO:tasks.cephadm:Archiving logs... 2024-03-11T21:20:57.356 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi001/log 2024-03-11T21:20:57.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:20:57.474 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi017/log 2024-03-11T21:20:57.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:20:57.564 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi204/log 2024-03-11T21:20:57.565 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:20:57.648 INFO:tasks.cephadm:Removing cluster... 2024-03-11T21:20:57.648 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --force 2024-03-11T21:20:57.831 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting cluster with fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:20:57.945 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --force 2024-03-11T21:20:58.131 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:20:58.243 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 415704e2-dfec-11ee-95c8-87774f69a715 --force 2024-03-11T21:20:58.425 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 415704e2-dfec-11ee-95c8-87774f69a715 2024-03-11T21:20:58.540 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T21:20:58.540 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:20:58.558 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:20:58.573 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:20:58.588 INFO:tasks.cephadm:Teardown complete 2024-03-11T21:20:58.589 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T21:20:58.600 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi001:/dev/vg_nvme/lv_1... 2024-03-11T21:20:58.600 DEBUG:teuthology.orchestra.run.smithi001:> sudo nvme disconnect -n lv_1 2024-03-11T21:20:58.742 INFO:teuthology.orchestra.run.smithi001.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:20:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:58.744 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi001:/dev/vg_nvme/lv_2... 2024-03-11T21:20:58.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo nvme disconnect -n lv_2 2024-03-11T21:20:58.887 INFO:teuthology.orchestra.run.smithi001.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:20:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:58.889 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi001:/dev/vg_nvme/lv_3... 2024-03-11T21:20:58.889 DEBUG:teuthology.orchestra.run.smithi001:> sudo nvme disconnect -n lv_3 2024-03-11T21:20:59.059 INFO:teuthology.orchestra.run.smithi001.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:20:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:59.061 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi001:/dev/vg_nvme/lv_4... 2024-03-11T21:20:59.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo nvme disconnect -n lv_4 2024-03-11T21:20:59.201 INFO:teuthology.orchestra.run.smithi001.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:20:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:59.203 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-03-11T21:20:59.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/scratch_devs 2024-03-11T21:20:59.230 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi017:/dev/vg_nvme/lv_1... 2024-03-11T21:20:59.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo nvme disconnect -n lv_1 2024-03-11T21:20:59.369 INFO:teuthology.orchestra.run.smithi017.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:20:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:59.371 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi017:/dev/vg_nvme/lv_2... 2024-03-11T21:20:59.371 DEBUG:teuthology.orchestra.run.smithi017:> sudo nvme disconnect -n lv_2 2024-03-11T21:20:59.529 INFO:teuthology.orchestra.run.smithi017.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:20:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:59.531 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi017:/dev/vg_nvme/lv_3... 2024-03-11T21:20:59.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo nvme disconnect -n lv_3 2024-03-11T21:20:59.688 INFO:teuthology.orchestra.run.smithi017.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:20:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:59.689 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi017:/dev/vg_nvme/lv_4... 2024-03-11T21:20:59.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo nvme disconnect -n lv_4 2024-03-11T21:20:59.827 INFO:teuthology.orchestra.run.smithi017.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:20:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:20:59.828 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-11T21:20:59.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/scratch_devs 2024-03-11T21:20:59.857 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_1... 2024-03-11T21:20:59.857 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_1 2024-03-11T21:21:00.007 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:21:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.009 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_2... 2024-03-11T21:21:00.009 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_2 2024-03-11T21:21:00.151 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:21:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.153 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_3... 2024-03-11T21:21:00.154 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_3 2024-03-11T21:21:00.349 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:21:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.351 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi204:/dev/vg_nvme/lv_4... 2024-03-11T21:21:00.351 DEBUG:teuthology.orchestra.run.smithi204:> sudo nvme disconnect -n lv_4 2024-03-11T21:21:00.519 INFO:teuthology.orchestra.run.smithi204.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:21:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.521 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-11T21:21:00.521 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/scratch_devs 2024-03-11T21:21:00.548 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T21:21:00.558 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T21:21:00.558 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:21:00.561 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:21:00.564 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:21:00.574 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:21:00.578 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:21:00.579 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:21:00.579 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-03-11T21:21:00.579 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +3126us[+3126us] +/- 111ms 2024-03-11T21:21:00.579 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -1967us[-1970us] +/- 62ms 2024-03-11T21:21:00.579 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +329us[ +329us] +/- 62ms 2024-03-11T21:21:00.580 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:21:00.582 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:21:00.582 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-03-11T21:21:00.582 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +3127us[+3127us] +/- 111ms 2024-03-11T21:21:00.582 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -1976us[-1946us] +/- 62ms 2024-03-11T21:21:00.583 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +342us[ +342us] +/- 62ms 2024-03-11T21:21:00.583 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:21:00.602 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:21:00.606 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:21:00.606 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-11T21:21:00.607 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +3147us[+3147us] +/- 111ms 2024-03-11T21:21:00.607 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -1948us[-1941us] +/- 62ms 2024-03-11T21:21:00.607 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +348us[ +348us] +/- 62ms 2024-03-11T21:21:00.607 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:21:00.607 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T21:21:00.618 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T21:21:00.619 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T21:21:00.630 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:21:00.707 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:21:00.782 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:21:00.851 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:21:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:21:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.915 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:21:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:00.945 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T21:21:00.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T21:21:00.964 INFO:teuthology.task.internal:Duration was 699.302148 seconds 2024-03-11T21:21:00.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T21:21:00.973 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T21:21:00.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:21:00.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:21:00.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:21:01.017 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:21:01.022 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:21:01.028 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:21:01.496 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T21:21:01.497 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-03-11T21:21:01.497 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:21:01.557 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-03-11T21:21:01.558 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:21:01.582 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-11T21:21:01.582 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:21:01.603 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T21:21:01.603 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:21:01.606 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:21:01.624 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:21:01.672 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T21:21:01.672 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:21:01.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:21:01.697 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:21:01.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T21:21:01.880 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T21:21:01.880 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:21:01.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:21:01.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:21:01.961 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T21:21:01.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:21:01.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:21:01.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:21:01.997 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-03-11T21:21:01.999 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-03-11T21:21:02.026 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-11T21:21:02.050 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:21:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:02.073 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:21:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:02.087 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:21:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:21:02.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T21:21:02.115 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T21:21:02.116 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi001 2024-03-11T21:21:02.117 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:21:02.155 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi017 2024-03-11T21:21:02.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:21:02.192 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800/remote/smithi204 2024-03-11T21:21:02.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:21:02.233 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T21:21:02.233 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:21:02.238 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:21:02.241 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:21:02.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T21:21:02.293 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T21:21:02.293 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T21:21:02.302 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T21:21:02.302 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:21:02.304 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:21:02.306 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:21:02.316 INFO:teuthology.orchestra.run.smithi001.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:21 /home/ubuntu/cephtest 2024-03-11T21:21:02.318 INFO:teuthology.orchestra.run.smithi017.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:21 /home/ubuntu/cephtest 2024-03-11T21:21:02.336 INFO:teuthology.orchestra.run.smithi204.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:21 /home/ubuntu/cephtest 2024-03-11T21:21:02.337 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T21:21:02.349 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T21:21:02.407 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T21:21:02.433 DEBUG:teuthology.nuke:shortname: smithi001 2024-03-11T21:21:02.433 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:21:02.456 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:35.959325', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLjmPxKzqo5hW0t7uLGuSiT1cdqGWUjCaULDOw7MY7BoFDoFDFVZ1wf1QRkCm96r/HlE2jCalpYPDGENTJsQ+Eo='} 2024-03-11T21:21:02.461 DEBUG:teuthology.nuke:shortname: smithi017 2024-03-11T21:21:02.461 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:21:02.483 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-11T21:21:02.484 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:21:02.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:35.960748', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFaTxXxg/pq5ZBgqILyMobvDvddS5eTBGliaLkKLYgwl9pfk0K0+xI7/SmR1UNrnwa74q/6Iu6Ec4qTuIDz5uHU='} 2024-03-11T21:21:02.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:35.957824', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7MdCpn/e6IhZZobHWhkTFb6n9x/GDhgF6p8CHFyvpp8Re9PqSahYU2win+rD6ewfuy1NPNsIxWeecTRxmuMi4='} 2024-03-11T21:21:02.519 INFO:teuthology.orchestra.console.smithi001:Power off 2024-03-11T21:21:02.520 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:21:02.546 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-03-11T21:21:02.546 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:02.561 INFO:teuthology.orchestra.console.smithi017:Power off 2024-03-11T21:21:02.562 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:21:02.573 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-11T21:21:02.583 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2024-03-11T21:21:02.583 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:02.598 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-11T21:21:02.599 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:21:02.606 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-03-11T21:21:02.620 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-11T21:21:02.621 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:02.641 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-11T21:21:06.574 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:06.607 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:06.642 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:06.685 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-03-11T21:21:06.717 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-03-11T21:21:06.751 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-11T21:21:10.686 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:10.719 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:10.752 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:21:10.797 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-03-11T21:21:10.829 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2024-03-11T21:21:10.862 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-11T21:21:10.898 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-03-11T21:21:10.930 INFO:teuthology.orchestra.console.smithi017:Power off completed 2024-03-11T21:21:10.963 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-11T21:21:11.068 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} duration: 699.3021476268768 failure_reason: '"2024-03-11T21:17:08.454036+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T21:21:11.068 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:21:11.157 INFO:teuthology.run:FAIL