2024-03-24T11:06:38.082 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T11:06:38.082 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895 --owner scheduled_teuthology@teuthology --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --verbose --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/orig.config.yaml 2024-03-24T11:06:38.110 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:06:38.190 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617895' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiEBlVUeWGsLpzLiMTCPg7Pe57C5FcD2O5TVEBkD4nHRg1g/VXm5ITHsaJLFdW4Z4Capwzb8tdTOKoKX9zqEGs= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7oM0a+vJpqRFpA/47Ap6v19P8bGX4Lz2KLOJwTJ4DXdB5+F7m+AyUyChWOEuv6FBJ4zNdfWD2+g9cu2Sm5kcY= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBML+x12VA0DdZ+ylW/KTlOj+3dIz2f/aTzdHblLXpubYJB+PVN+rWsUENRFARcw/JSPirxjvRK07har/RzmlnAE= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T11:06:38.190 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T11:06:38.191 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T11:06:38.191 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T11:06:38.193 INFO:teuthology.task.internal:Checking packages... 2024-03-24T11:06:38.219 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T11:06:38.219 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:06:38.219 INFO:teuthology.packaging:ref: None 2024-03-24T11:06:38.220 INFO:teuthology.packaging:tag: None 2024-03-24T11:06:38.220 INFO:teuthology.packaging:branch: squid 2024-03-24T11:06:38.220 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:06:38.220 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T11:06:38.450 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T11:06:38.452 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T11:06:38.458 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T11:06:38.459 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T11:06:38.464 INFO:teuthology.task.internal:Saving configuration 2024-03-24T11:06:38.477 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T11:06:38.487 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:06:38.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:01:20.713104', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiEBlVUeWGsLpzLiMTCPg7Pe57C5FcD2O5TVEBkD4nHRg1g/VXm5ITHsaJLFdW4Z4Capwzb8tdTOKoKX9zqEGs='} 2024-03-24T11:06:38.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:01:20.714262', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7oM0a+vJpqRFpA/47Ap6v19P8bGX4Lz2KLOJwTJ4DXdB5+F7m+AyUyChWOEuv6FBJ4zNdfWD2+g9cu2Sm5kcY='} 2024-03-24T11:06:38.559 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:01:20.711687', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBML+x12VA0DdZ+ylW/KTlOj+3dIz2f/aTzdHblLXpubYJB+PVN+rWsUENRFARcw/JSPirxjvRK07har/RzmlnAE='} 2024-03-24T11:06:38.559 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T11:06:38.566 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T11:06:38.566 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-24T11:06:38.566 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-24T11:06:38.566 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T11:06:38.673 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c52c21f70>, signals=[15]) 2024-03-24T11:06:38.674 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T11:06:38.680 INFO:teuthology.task.internal:Opening connections... 2024-03-24T11:06:38.680 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-03-24T11:06:38.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:06:38.775 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-03-24T11:06:38.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:06:38.864 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T11:06:38.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:06:38.953 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T11:06:38.960 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-03-24T11:06:38.978 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-03-24T11:06:38.978 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-03-24T11:06:39.034 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:06:39.035 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:06:39.036 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:06:39.036 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-03-24T11:06:39.060 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-03-24T11:06:39.078 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-03-24T11:06:39.078 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-03-24T11:06:39.134 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-03-24T11:06:39.134 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:06:39.135 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:06:39.136 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:06:39.136 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-03-24T11:06:39.160 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-03-24T11:06:39.178 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-03-24T11:06:39.178 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-03-24T11:06:39.235 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T11:06:39.236 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T11:06:39.236 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-03-24T11:06:39.260 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T11:06:39.270 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T11:06:39.276 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T11:06:39.276 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:06:39.279 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:06:39.281 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-03-24T11:06:39.297 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T11:06:39.303 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T11:06:39.303 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:06:39.336 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:06:39.338 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-03-24T11:06:39.441 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T11:06:39.521 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T11:06:39.535 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T11:06:39.535 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T11:06:39.536 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:06:39.536 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:06:39.537 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T11:06:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:06:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:06:39.554 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-24T11:06:39.555 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-03-24T11:06:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:06:39.556 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-24T11:06:39.611 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:06:39.612 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-03-24T11:06:39.612 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-03-24T11:06:39.613 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:06:39.613 INFO:teuthology.task.kernel:Running kernel on smithi184: 5.14.0-361.el9.x86_64 2024-03-24T11:06:39.613 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-03-24T11:06:39.615 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-361.el9.x86_64 2024-03-24T11:06:39.615 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-361.el9.x86_64 2024-03-24T11:06:39.615 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-03-24T11:06:41.635 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T11:06:41.745 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-24T11:06:41.802 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T11:06:43.068 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-24T11:06:43.129 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-24T11:06:43.461 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-24T11:06:43.994 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T11:06:44.185 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 23 MB/s | 8.0 MB 00:00 2024-03-24T11:06:45.569 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-24T11:06:45.885 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-24T11:06:47.612 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-24T11:06:50.021 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.1 MB 00:00 2024-03-24T11:06:51.546 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-24T11:06:51.562 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-24T11:06:53.557 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T11:06:53.863 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-24T11:06:55.856 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T11:06:58.406 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T11:07:00.511 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:07:00.511 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:07:00.599 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-24T11:07:00.607 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-24T11:07:00.607 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-24T11:07:00.607 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-24T11:07:00.607 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-24T11:07:00.607 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:07:00.607 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-03-24T11:07:00.608 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:00.609 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 76 M 2024-03-24T11:07:00.609 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-03-24T11:07:00.609 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-24T11:07:00.848 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-24T11:07:00.901 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-24T11:07:00.947 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 229 kB/s | 44 kB 00:00 2024-03-24T11:07:01.556 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-24T11:07:02.139 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 24 MB/s | 33 MB 00:01 2024-03-24T11:07:02.456 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 18 MB/s | 27 MB 00:01 2024-03-24T11:07:02.456 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:02.457 INFO:teuthology.orchestra.run.smithi186.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-24T11:07:03.005 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:07:03.005 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:07:03.085 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:07:03.095 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-24T11:07:03.096 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-24T11:07:03.096 INFO:teuthology.orchestra.run.smithi184.stdout:Install 4 Packages 2024-03-24T11:07:03.096 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:07:03.096 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 76 M 2024-03-24T11:07:03.096 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 123 M 2024-03-24T11:07:03.096 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-24T11:07:03.115 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-24T11:07:03.482 INFO:teuthology.orchestra.run.smithi184.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 130 kB/s | 44 kB 00:00 2024-03-24T11:07:03.510 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-24T11:07:03.511 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-24T11:07:04.516 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-24T11:07:04.516 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-24T11:07:05.035 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-24T11:07:06.469 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-24T11:07:07.000 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-24T11:07:07.601 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:10.062 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 21 MB 00:03 2024-03-24T11:07:10.478 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:10.617 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:15.003 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-24T11:07:15.338 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:15.375 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:16.772 INFO:teuthology.orchestra.run.smithi184.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.1 MB/s | 15 MB 00:13 2024-03-24T11:07:17.127 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T11:07:17.127 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T11:07:17.229 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:07:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-03-24T11:07:17.240 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:07:17.240 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 76 M 2024-03-24T11:07:17.240 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 123 M 2024-03-24T11:07:17.240 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-24T11:07:17.871 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 126 kB/s | 44 kB 00:00 2024-03-24T11:07:20.113 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:27.920 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.4 MB/s | 15 MB 00:10 2024-03-24T11:07:32.503 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.9 MB/s | 27 MB 00:14 2024-03-24T11:07:32.737 INFO:teuthology.orchestra.run.smithi184.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 961 kB/s | 27 MB 00:29 2024-03-24T11:07:40.728 INFO:teuthology.orchestra.run.smithi184.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 911 kB/s | 33 MB 00:37 2024-03-24T11:07:40.729 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:40.729 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.0 MB/s | 76 MB 00:37 2024-03-24T11:07:41.385 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-24T11:07:41.786 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-24T11:07:41.786 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-24T11:07:42.418 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.3 MB/s | 33 MB 00:24 2024-03-24T11:07:42.419 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:42.419 INFO:teuthology.orchestra.run.smithi189.stdout:Total 3.0 MB/s | 76 MB 00:25 2024-03-24T11:07:42.770 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-24T11:07:42.770 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-24T11:07:43.089 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-24T11:07:43.473 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-24T11:07:43.473 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-24T11:07:44.477 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-24T11:07:44.478 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-24T11:07:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-24T11:07:45.916 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:07:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:46.964 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-24T11:07:47.918 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:07:48.002 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:48.822 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:48.958 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:49.283 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:49.283 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:07:49.283 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:49.328 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:49.721 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:49.721 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:07:49.721 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:49.721 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:50.136 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-24T11:07:50.485 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:07:50.622 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:07:50.622 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:07:50.623 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:07:50.623 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:07:50.623 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:07:50.623 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T11:07:50.623 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:07:50.623 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:07:50.703 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:07:50.704 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:07:50.704 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-03-24T11:07:50.739 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-03-24T11:07:50.739 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-03-24T11:07:50.739 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-03-24T11:07:50.739 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-03-24T11:07:50.739 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:50.740 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:07:50.741 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:07:50.741 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:07:50.741 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-03-24T11:07:50.741 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:07:50.741 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:07:50.742 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:07:50.742 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:07:50.742 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:07:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:07:50.924 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:51.337 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-03-24T11:07:52.669 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:07:52.708 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-03-24T11:07:52.710 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-24T11:07:52.735 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.DbnJWJYK9J 2024-03-24T11:07:52.735 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DbnJWJYK9J 2024-03-24T11:07:52.804 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.DbnJWJYK9J 2024-03-24T11:07:52.945 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.DbnJWJYK9J is 6KB 2024-03-24T11:07:52.993 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.DbnJWJYK9J 2024-03-24T11:07:53.007 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:07:53.073 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:07:53.073 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:07:53.073 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:07:53.073 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:07:53.074 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:07:53.230 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-03-24T11:07:53.690 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:53.724 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:55.627 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:07:55.672 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:07:58.468 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:00.442 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:23.233 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T11:08:23.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-24T11:08:23.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:08:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:27.748 INFO:teuthology.orchestra.run.smithi184.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:08:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:08:27.793 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:08:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:08:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:08:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:28.518 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:08:28.519 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-24T11:08:28.869 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:08:28.962 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:08:28.962 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:08:28.962 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:08:28.962 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:08:28.962 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:08:28.963 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T11:08:28.963 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:08:28.963 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:08:29.043 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:08:29.044 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:08:29.044 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-03-24T11:08:29.078 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-03-24T11:08:29.078 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-03-24T11:08:29.078 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.06 2024-03-24T11:08:29.078 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 68.el9 2024-03-24T11:08:29.078 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-03-24T11:08:29.078 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Group : Unspecified 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 8264254 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : builder@centos.org 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:08:29.079 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:08:29.080 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-03-24T11:08:29.080 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:08:29.080 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:08:29.081 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:08:29.081 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:08:29.081 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:08:29.706 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-03-24T11:08:29.755 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:29.755 INFO:teuthology.orchestra.run.smithi189.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:08:29.755 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:08:29.799 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:30.225 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:30.226 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:08:30.226 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:08:30.226 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:08:30.665 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:08:30.665 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:08:30.666 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-24T11:08:30.994 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:08:31.106 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:08:31.106 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:08:31.107 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:08:31.107 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:08:31.107 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:08:31.107 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-24T11:08:31.107 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:08:31.107 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:08:31.117 INFO:teuthology.orchestra.run.smithi184.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:08:31.168 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-03-24T11:08:31.170 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-03-24T11:08:31.186 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.cVRnUcPKdY 2024-03-24T11:08:31.189 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cVRnUcPKdY 2024-03-24T11:08:31.191 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:08:31.191 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:08:31.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-03-24T11:08:31.225 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-03-24T11:08:31.225 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-03-24T11:08:31.225 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-03-24T11:08:31.226 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:08:31.227 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:08:31.228 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:08:31.228 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:08:31.228 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:08:31.255 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.cVRnUcPKdY 2024-03-24T11:08:31.395 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.cVRnUcPKdY is 6KB 2024-03-24T11:08:31.443 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.cVRnUcPKdY 2024-03-24T11:08:31.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:08:31.531 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:08:31.531 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:08:31.531 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:08:31.531 INFO:teuthology.orchestra.run.smithi184.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:08:31.533 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:08:31.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-03-24T11:08:31.831 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-03-24T11:08:33.279 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:08:33.310 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-03-24T11:08:33.312 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-03-24T11:08:33.327 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.R5DEBl9YA6 2024-03-24T11:08:33.327 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R5DEBl9YA6 2024-03-24T11:08:33.397 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.R5DEBl9YA6 2024-03-24T11:08:33.537 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.R5DEBl9YA6 is 6KB 2024-03-24T11:08:33.557 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.R5DEBl9YA6 2024-03-24T11:08:33.571 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:08:33.636 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:08:33.636 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:08:33.636 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:08:33.636 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:08:33.637 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:08:33.786 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-03-24T11:08:57.763 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-24T11:09:01.686 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T11:09:01.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-24T11:09:01.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:09:03.791 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-24T11:09:03.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:09:03.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:09:06.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-24T11:09:06.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:09:25.154 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-24T11:09:37.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-24T11:09:37.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:09:38.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-24T11:09:39.376 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-24T11:09:40.024 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-24T11:09:40.024 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:09:40.024 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-24T11:09:40.038 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:09:40.039 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:09:40.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:09:40.039 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:09:41.039 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T11:09:41.040 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:09:41.131 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-03-24T11:09:41.179 DEBUG:teuthology.parallel:result is None 2024-03-24T11:09:47.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:09:47.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:10:01.690 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:10:10.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-24T11:10:10.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:10:12.093 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-03-24T11:10:12.634 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-24T11:10:12.634 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:10:12.634 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-03-24T11:10:12.648 INFO:teuthology.orchestra.run.smithi184.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:10:12.648 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:10:12.648 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:10:12.648 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:10:13.649 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T11:10:13.649 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:10:13.712 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-03-24T11:10:13.768 DEBUG:teuthology.parallel:result is None 2024-03-24T11:10:47.467 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:10:59.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:10:59.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:10:59.814 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-24T11:11:00.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:11:00.207 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:11:00.208 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-24T11:11:00.224 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:11:00.224 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:11:00.224 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:11:00.225 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:11:01.225 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-24T11:11:01.225 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:11:01.302 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-03-24T11:11:01.339 DEBUG:teuthology.parallel:result is None 2024-03-24T11:11:01.340 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:11:01.347 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:11:01.348 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:11:01.351 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:11:01.354 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:11:01.372 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:11:01.378 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:11:01.386 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:11:01.386 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:11:01.409 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:11:01.413 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:11:01.531 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:11:01.538 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:11:01.538 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:11:01.541 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:11:01.543 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:11:01.576 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:11:01.577 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:11:01.585 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:11:01.590 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:11:01.592 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:11:01.600 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:11:01.601 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:11:01.609 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:11:01.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:11:01.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:11:01.637 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:11:01.666 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:11:01.676 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:11:01.676 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:11:01.706 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:11:01.709 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:11:01.726 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:11:01.801 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:11:01.855 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:11:01.933 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:11:01.980 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:11:01.980 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:11:02.048 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:11:02.095 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:11:02.161 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:11:02.199 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:11:02.250 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:11:02.250 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:11:02.319 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:11:02.357 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:11:02.427 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:11:02.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:11:02.514 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:11:02.514 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:11:02.582 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-03-24T11:11:02.585 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-03-24T11:11:02.587 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-03-24T11:11:02.645 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:11:02.645 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:11:02.669 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:11:03.082 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:11:03.091 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:11:03.091 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:11:03.102 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:11:03.110 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:11:03.111 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-03-24T11:11:03.170 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-03-24T11:11:03.172 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-03-24T11:11:03.236 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-03-24T11:11:03.238 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-03-24T11:11:03.305 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-03-24T11:11:03.307 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:11:03.307 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-03-24T11:11:03.325 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-03-24T11:11:03.325 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-03-24T11:11:03.346 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-03-24T11:11:03.347 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-03-24T11:11:03.378 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-03-24T11:11:03.378 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi184.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:11:03.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:11:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:11:03.407 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:11:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:11:03.436 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:11:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:11:03.464 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:11:03.465 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:11:03.502 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:11:03.532 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:11:03.571 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:11:03.581 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T11:11:03.586 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T11:11:03.719 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:11:03.735 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:11:03.737 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi184.front.sepia.ceph.com,smithi186.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:15:15.398 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi184.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-03-24T11:15:15.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-24T11:15:15.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:15:15.491 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-03-24T11:15:15.568 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-03-24T11:15:15.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-24T11:15:15.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:15:15.657 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-24T11:15:15.748 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-24T11:15:15.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:15:15.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:15:15.837 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-24T11:15:15.917 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T11:15:15.917 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:15:15.928 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:15:15.928 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:15:15.928 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:15:15.931 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:15:15.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:15:15.933 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:15:15.933 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:15:15.960 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:15:15.962 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:15:15.976 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:15:15.979 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:15:15.988 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:15:16.005 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:15:16.005 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-03-24T11:15:16.008 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-03-24T11:15:16.019 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-03-24T11:15:16.022 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-03-24T11:15:16.034 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-03-24T11:15:16.035 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:15:16.038 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:15:16.048 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-03-24T11:15:16.052 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:15:16.055 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:15:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:15:16.080 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:15:16.102 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:15:16.105 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:15:16.105 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-03-24T11:15:16.105 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.105 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.106 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.106 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.106 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:15:16.110 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:15:16.110 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-24T11:15:16.110 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.110 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.110 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.110 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.128 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:15:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:15:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-24T11:15:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.131 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:15:16.132 INFO:teuthology.run_tasks:Running task pexec... 2024-03-24T11:15:16.141 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-24T11:15:16.142 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi184.front.sepia.ceph.com 2024-03-24T11:15:16.142 DEBUG:teuthology.orchestra.run.smithi184:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:15:16.142 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi186.front.sepia.ceph.com 2024-03-24T11:15:16.142 DEBUG:teuthology.orchestra.run.smithi186:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:15:16.143 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T11:15:16.143 DEBUG:teuthology.orchestra.run.smithi189:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T11:15:16.734 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:49 ago on Sun 24 Mar 2024 11:14:27 AM UTC. 2024-03-24T11:15:16.735 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:48 ago on Sun 24 Mar 2024 11:14:28 AM UTC. 2024-03-24T11:15:16.760 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:49 ago on Sun 24 Mar 2024 11:14:27 AM UTC. 2024-03-24T11:15:16.882 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-24T11:15:16.883 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-03-24T11:15:16.893 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-03-24T11:15:16.894 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-03-24T11:15:16.895 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-03-24T11:15:16.895 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:15:16.897 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.1 M 2024-03-24T11:15:16.897 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 9.8 M 2024-03-24T11:15:16.897 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-03-24T11:15:16.908 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-24T11:15:16.910 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-24T11:15:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:15:16.913 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 3.1 M 2024-03-24T11:15:16.913 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 9.8 M 2024-03-24T11:15:16.913 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-24T11:15:16.922 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.1 M 2024-03-24T11:15:16.922 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 9.8 M 2024-03-24T11:15:16.922 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-24T11:15:17.231 INFO:teuthology.orchestra.run.smithi189.stdout:runc-1.1.12-2.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-03-24T11:15:17.231 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:15:17.232 INFO:teuthology.orchestra.run.smithi189.stdout:Total 10 MB/s | 3.1 MB 00:00 2024-03-24T11:15:17.268 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-24T11:15:17.273 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-24T11:15:17.273 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-24T11:15:17.353 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-24T11:15:17.354 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-24T11:15:17.579 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-24T11:15:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:runc-1.1.12-2.el9.x86_64.rpm 6.1 MB/s | 3.1 MB 00:00 2024-03-24T11:15:17.680 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:15:17.681 INFO:teuthology.orchestra.run.smithi184.stdout:Total 4.0 MB/s | 3.1 MB 00:00 2024-03-24T11:15:17.692 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:15:17.717 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-03-24T11:15:17.722 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-03-24T11:15:17.722 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-03-24T11:15:17.803 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-03-24T11:15:17.803 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-03-24T11:15:18.004 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-03-24T11:15:18.091 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:15:18.216 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:15:18.541 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:15:18.541 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:15:18.541 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-24T11:15:18.541 INFO:teuthology.orchestra.run.smithi189.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:15:18.541 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:15:18.541 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-24T11:15:18.689 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:15:18.763 DEBUG:teuthology.parallel:result is None 2024-03-24T11:15:18.981 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:15:18.982 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:15:18.982 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-03-24T11:15:18.982 INFO:teuthology.orchestra.run.smithi184.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:15:18.982 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:15:18.982 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-03-24T11:15:19.261 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:19.554 INFO:teuthology.orchestra.run.smithi186.stdout:runc-1.1.12-2.el9.x86_64.rpm 51 kB/s | 3.1 MB 01:02 2024-03-24T11:16:19.555 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:16:19.555 INFO:teuthology.orchestra.run.smithi186.stdout:Total 51 kB/s | 3.1 MB 01:02 2024-03-24T11:16:19.589 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-24T11:16:19.596 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-24T11:16:19.596 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-24T11:16:19.676 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-24T11:16:19.677 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-24T11:16:19.910 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-24T11:16:19.997 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:16:20.457 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:16:20.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T11:16:20.751 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:16:20.751 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-24T11:16:20.752 INFO:teuthology.orchestra.run.smithi186.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T11:16:20.752 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-24T11:16:20.752 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-24T11:16:20.990 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:20.990 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-24T11:16:21.002 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-24T11:16:21.002 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:16:21.002 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:16:21.021 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:16:21.021 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 2024-03-24T11:16:21.078 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:15:14.384894488 +0000 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:14:51.806374165 +0000 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:14:51.806374165 +0000 2024-03-24T11:16:21.079 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:14:51.806374165 +0000 2024-03-24T11:16:21.079 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:16:21.149 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:16:21.149 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:16:21.149 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000237923 s, 2.2 MB/s 2024-03-24T11:16:21.151 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:16:21.209 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:15:14.384894488 +0000 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:14:52.150366857 +0000 2024-03-24T11:16:21.265 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:14:52.150366857 +0000 2024-03-24T11:16:21.266 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:14:52.150366857 +0000 2024-03-24T11:16:21.266 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:16:21.332 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:16:21.332 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:16:21.332 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000194019 s, 2.6 MB/s 2024-03-24T11:16:21.333 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:16:21.391 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 2024-03-24T11:16:21.448 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:16:21.448 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:21.448 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T11:16:21.448 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:21.448 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:21.449 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:15:14.384894488 +0000 2024-03-24T11:16:21.449 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:14:52.497359485 +0000 2024-03-24T11:16:21.449 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:14:52.497359485 +0000 2024-03-24T11:16:21.449 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:14:52.497359485 +0000 2024-03-24T11:16:21.449 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:16:21.514 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:16:21.514 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:16:21.514 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000193222 s, 2.6 MB/s 2024-03-24T11:16:21.516 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:16:21.572 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 2024-03-24T11:16:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:16:21.627 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:21.627 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-24T11:16:21.627 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:21.627 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:21.628 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:15:14.384894488 +0000 2024-03-24T11:16:21.628 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:14:52.835352305 +0000 2024-03-24T11:16:21.628 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:14:52.835352305 +0000 2024-03-24T11:16:21.628 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:14:52.835352305 +0000 2024-03-24T11:16:21.628 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:16:21.692 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:16:21.693 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:16:21.693 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000194124 s, 2.6 MB/s 2024-03-24T11:16:21.694 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:16:21.750 DEBUG:teuthology.orchestra.run.smithi184:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T11:16:21.890 INFO:teuthology.orchestra.run.smithi184.stdout:loop 2024-03-24T11:16:21.892 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_1... 2024-03-24T11:16:21.892 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T11:16:21.929 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-03-24T11:16:21.967 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_11 2024-03-24T11:16:22.008 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_2... 2024-03-24T11:16:22.008 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T11:16:22.085 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-03-24T11:16:22.122 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_21 2024-03-24T11:16:22.150 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_3... 2024-03-24T11:16:22.151 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T11:16:22.229 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-03-24T11:16:22.266 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_31 2024-03-24T11:16:22.301 INFO:tasks.nvme_loop:Connecting nvme_loop smithi184:/dev/vg_nvme/lv_4... 2024-03-24T11:16:22.301 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T11:16:22.378 INFO:teuthology.orchestra.run.smithi184.stdout:1 2024-03-24T11:16:22.415 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_41 2024-03-24T11:16:22.444 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:16:22.444 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:16:22.499 DEBUG:teuthology.orchestra.run.smithi184:> sudo nvme list 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme0n1 CVFT626500CE400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme1n1 6f0cdbb475a75095928a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme2n1 34685be692930456d029 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme3n1 ac26ed7a7a1c4e45d3ca Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:22.565 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/nvme4n1 648a2b1fd71b8245e591 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:22.566 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:16:22.566 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:16:22.566 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/scratch_devs 2024-03-24T11:16:22.633 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:16:22.633 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:16:22.650 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:16:22.651 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-03-24T11:16:22.707 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:16:22.707 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:22.707 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T11:16:22.708 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:22.708 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:22.708 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:15:14.401503291 +0000 2024-03-24T11:16:22.708 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:14:52.930884285 +0000 2024-03-24T11:16:22.708 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:14:52.930884285 +0000 2024-03-24T11:16:22.708 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:14:52.930884285 +0000 2024-03-24T11:16:22.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:16:22.774 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:16:22.775 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:16:22.775 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000268148 s, 1.9 MB/s 2024-03-24T11:16:22.776 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:16:22.832 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-03-24T11:16:22.888 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:16:22.888 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:22.888 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T11:16:22.889 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:22.889 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:22.889 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:15:14.401503291 +0000 2024-03-24T11:16:22.889 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:14:53.277878113 +0000 2024-03-24T11:16:22.889 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:14:53.277878113 +0000 2024-03-24T11:16:22.889 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:14:53.277878113 +0000 2024-03-24T11:16:22.889 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:16:22.954 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:16:22.954 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:16:22.955 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000197582 s, 2.6 MB/s 2024-03-24T11:16:22.956 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:16:23.013 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:15:14.401503291 +0000 2024-03-24T11:16:23.068 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:14:53.638871693 +0000 2024-03-24T11:16:23.069 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:14:53.638871693 +0000 2024-03-24T11:16:23.069 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:14:53.638871693 +0000 2024-03-24T11:16:23.069 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:16:23.134 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:16:23.134 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:16:23.134 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000222512 s, 2.3 MB/s 2024-03-24T11:16:23.135 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:16:23.190 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-03-24T11:16:23.245 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:16:23.245 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:23.245 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T11:16:23.246 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:23.246 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:23.246 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:15:14.401503291 +0000 2024-03-24T11:16:23.246 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:14:53.991865415 +0000 2024-03-24T11:16:23.246 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:14:53.991865415 +0000 2024-03-24T11:16:23.246 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:14:53.991865415 +0000 2024-03-24T11:16:23.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:16:23.310 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:16:23.310 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:16:23.311 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000194775 s, 2.6 MB/s 2024-03-24T11:16:23.312 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:16:23.367 DEBUG:teuthology.orchestra.run.smithi186:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T11:16:23.517 INFO:teuthology.orchestra.run.smithi186.stdout:loop 2024-03-24T11:16:23.519 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_1... 2024-03-24T11:16:23.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T11:16:23.557 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-03-24T11:16:23.594 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_11 2024-03-24T11:16:23.624 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_2... 2024-03-24T11:16:23.624 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T11:16:23.705 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-03-24T11:16:23.743 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_21 2024-03-24T11:16:23.771 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_3... 2024-03-24T11:16:23.771 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T11:16:23.850 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-03-24T11:16:23.887 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_31 2024-03-24T11:16:23.927 INFO:tasks.nvme_loop:Connecting nvme_loop smithi186:/dev/vg_nvme/lv_4... 2024-03-24T11:16:23.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T11:16:24.007 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-03-24T11:16:24.044 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/vg_nvme/lv_41 2024-03-24T11:16:24.072 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:16:24.073 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:16:24.133 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme list 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme0n1 CVFT623300CB400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme1n1 9e7fc2812de554b6985e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme2n1 016264dd34a73a59e4b4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme3n1 221fe5394eeda1ce3731 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:/dev/nvme4n1 cc71ecd1867a6b0a3ef5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:24.201 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:16:24.201 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:16:24.202 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/scratch_devs 2024-03-24T11:16:24.268 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:16:24.268 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:16:24.289 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:16:24.289 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:15:14.382363787 +0000 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:14:52.175784292 +0000 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:14:52.175784292 +0000 2024-03-24T11:16:24.347 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:14:52.175784292 +0000 2024-03-24T11:16:24.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:16:24.419 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:16:24.419 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:16:24.419 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000281126 s, 1.8 MB/s 2024-03-24T11:16:24.420 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:16:24.479 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-03-24T11:16:24.535 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:15:14.382363787 +0000 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:14:52.516777835 +0000 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:14:52.516777835 +0000 2024-03-24T11:16:24.536 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:14:52.516777835 +0000 2024-03-24T11:16:24.536 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:16:24.604 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:16:24.604 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:16:24.604 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000175148 s, 2.9 MB/s 2024-03-24T11:16:24.605 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:16:24.663 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-03-24T11:16:24.719 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:16:24.719 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:24.719 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T11:16:24.719 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:24.719 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:24.719 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:15:14.382363787 +0000 2024-03-24T11:16:24.720 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:14:52.855771416 +0000 2024-03-24T11:16:24.720 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:14:52.855771416 +0000 2024-03-24T11:16:24.720 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:14:52.855771416 +0000 2024-03-24T11:16:24.720 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:16:24.785 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:16:24.786 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:16:24.786 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000202558 s, 2.5 MB/s 2024-03-24T11:16:24.787 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:16:24.844 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-03-24T11:16:24.899 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:15:14.382363787 +0000 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:14:53.187765129 +0000 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:14:53.187765129 +0000 2024-03-24T11:16:24.900 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:14:53.187765129 +0000 2024-03-24T11:16:24.900 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:16:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:16:24.966 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:16:24.967 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000166927 s, 3.1 MB/s 2024-03-24T11:16:24.967 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:16:25.023 DEBUG:teuthology.orchestra.run.smithi189:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-24T11:16:25.161 INFO:teuthology.orchestra.run.smithi189.stdout:loop 2024-03-24T11:16:25.162 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_1... 2024-03-24T11:16:25.163 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-24T11:16:25.200 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:16:25.237 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_11 2024-03-24T11:16:25.267 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_2... 2024-03-24T11:16:25.267 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-24T11:16:25.345 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:16:25.382 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_21 2024-03-24T11:16:25.418 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_3... 2024-03-24T11:16:25.418 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-24T11:16:25.497 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:16:25.534 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_31 2024-03-24T11:16:25.562 INFO:tasks.nvme_loop:Connecting nvme_loop smithi189:/dev/vg_nvme/lv_4... 2024-03-24T11:16:25.563 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-24T11:16:25.640 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-24T11:16:25.677 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_41 2024-03-24T11:16:25.713 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:16:25.713 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:16:25.768 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme list 2024-03-24T11:16:25.834 INFO:teuthology.orchestra.run.smithi189.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-24T11:16:25.835 INFO:teuthology.orchestra.run.smithi189.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-24T11:16:25.835 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme0n1 CVFT6232004L400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-24T11:16:25.835 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme1n1 bcebc7a9d322a8561536 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:25.835 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme2n1 8b4ea99af3db48bd7ece Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:25.835 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme3n1 610a6d81b2400c77931b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:25.835 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/nvme4n1 492ff378e68fc379c85f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-24T11:16:25.836 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:16:25.836 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:16:25.836 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/scratch_devs 2024-03-24T11:16:25.903 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:16:26.001 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:16:26.001 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:16:26.001 INFO:tasks.cephadm:Cluster fsid is f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:16:26.002 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:16:26.002 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.184', 'mon.b': '172.21.15.186', 'mon.c': '172.21.15.189'} 2024-03-24T11:16:26.002 INFO:tasks.cephadm:First mon is mon.a on smithi184 2024-03-24T11:16:26.002 INFO:tasks.cephadm:First mgr is y 2024-03-24T11:16:26.002 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:16:26.002 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-03-24T11:16:26.035 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-03-24T11:16:26.068 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-03-24T11:16:26.096 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T11:16:26.096 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:16:26.319 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77704/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1667-gdb0330b1', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1667.gdb0330b1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 11:55:36.797747', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-24T11:16:26.539 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 db0330b1e4e2470d52b750e251e55a522b4f7d69 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:16:26.541 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm 2024-03-24T11:16:26.541 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:27.148 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:16 /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:27.148 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:27.742 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:16 /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:27.742 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:28.361 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 11:16 /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:28.362 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:28.381 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:28.400 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:16:28.430 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T11:16:28.430 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:16:28.433 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:16:28.442 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T11:16:29.207 INFO:teuthology.orchestra.run.smithi184.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:16:29.209 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:16:29.296 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout: "repo_digests": [ 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2024-03-24T11:17:00.043 INFO:teuthology.orchestra.run.smithi184.stdout:} 2024-03-24T11:17:04.198 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-24T11:17:04.198 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:17:04.198 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:17:04.199 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-03-24T11:17:04.199 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:17:04.199 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-03-24T11:17:04.199 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-24T11:17:04.735 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-24T11:17:04.735 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T11:17:04.736 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T11:17:04.736 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-03-24T11:17:04.736 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T11:17:04.736 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-03-24T11:17:04.736 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-24T11:17:04.762 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-03-24T11:17:04.796 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-03-24T11:17:04.825 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-03-24T11:17:04.857 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-03-24T11:17:04.888 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-03-24T11:17:04.915 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-03-24T11:17:04.944 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:17:04.945 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:17:04.946 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:17:04.946 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:17:04.946 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T11:17:04.946 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:17:04.947 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:17:04.963 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f43a2692-e9cf-11ee-95cd-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-24T11:17:04.963 DEBUG:teuthology.orchestra.run.smithi184:mon.a> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T11:17:05.006 DEBUG:teuthology.orchestra.run.smithi184:mgr.y> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T11:17:05.049 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:17:05.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.184 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:17:05.239 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:17:05.240 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', 'f43a2692-e9cf-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.184', '--skip-admin-label'] 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi184.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying podman|docker is present... 2024-03-24T11:17:05.289 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:17:05.290 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying lvm2 is present... 2024-03-24T11:17:05.290 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying time synchronization is in place... 2024-03-24T11:17:05.296 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:17:05.296 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:17:05.303 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:17:05.303 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-03-24T11:17:05.309 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout enabled 2024-03-24T11:17:05.314 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout active 2024-03-24T11:17:05.315 INFO:teuthology.orchestra.run.smithi184.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:17:05.315 INFO:teuthology.orchestra.run.smithi184.stdout:Repeating the final host check... 2024-03-24T11:17:05.338 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:17:05.338 INFO:teuthology.orchestra.run.smithi184.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:17:05.338 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl is present 2024-03-24T11:17:05.338 INFO:teuthology.orchestra.run.smithi184.stdout:lvcreate is present 2024-03-24T11:17:05.344 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:17:05.344 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:17:05.350 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:17:05.350 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-03-24T11:17:05.356 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout enabled 2024-03-24T11:17:05.362 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout active 2024-03-24T11:17:05.362 INFO:teuthology.orchestra.run.smithi184.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:17:05.362 INFO:teuthology.orchestra.run.smithi184.stdout:Host looks OK 2024-03-24T11:17:05.362 INFO:teuthology.orchestra.run.smithi184.stdout:Cluster fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:05.362 INFO:teuthology.orchestra.run.smithi184.stdout:Acquiring lock 139697667417136 on /run/cephadm/f43a2692-e9cf-11ee-95cd-87774f69a715.lock 2024-03-24T11:17:05.362 INFO:teuthology.orchestra.run.smithi184.stdout:Lock 139697667417136 acquired on /run/cephadm/f43a2692-e9cf-11ee-95cd-87774f69a715.lock 2024-03-24T11:17:05.363 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying IP 172.21.15.184 port 3300 ... 2024-03-24T11:17:05.363 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying IP 172.21.15.184 port 6789 ... 2024-03-24T11:17:05.363 INFO:teuthology.orchestra.run.smithi184.stdout:Base mon IP(s) is [172.21.15.184:3300, 172.21.15.184:6789], mon addrv is [v2:172.21.15.184:3300,v1:172.21.15.184:6789] 2024-03-24T11:17:05.366 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.184 metric 100 2024-03-24T11:17:05.366 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.184 metric 100 2024-03-24T11:17:05.369 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:17:05.369 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:17:05.369 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1763sec hoplimit 64 pref medium 2024-03-24T11:17:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:17:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T11:17:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:17:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:17:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7b8f/64 scope link 2024-03-24T11:17:05.371 INFO:teuthology.orchestra.run.smithi184.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:17:05.372 INFO:teuthology.orchestra.run.smithi184.stdout:Mon IP `172.21.15.184` is in CIDR network `172.21.0.0/20` 2024-03-24T11:17:05.372 INFO:teuthology.orchestra.run.smithi184.stdout:Mon IP `172.21.15.184` is in CIDR network `172.21.0.0/20` 2024-03-24T11:17:05.372 INFO:teuthology.orchestra.run.smithi184.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T11:17:05.372 INFO:teuthology.orchestra.run.smithi184.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:17:05.373 INFO:teuthology.orchestra.run.smithi184.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:17:05.828 INFO:teuthology.orchestra.run.smithi184.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T11:17:06.749 INFO:teuthology.orchestra.run.smithi184.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:17:06.749 INFO:teuthology.orchestra.run.smithi184.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T11:17:06.749 INFO:teuthology.orchestra.run.smithi184.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T11:17:07.670 INFO:teuthology.orchestra.run.smithi184.stdout:stat: stdout 167 167 2024-03-24T11:17:07.671 INFO:teuthology.orchestra.run.smithi184.stdout:Creating initial keys... 2024-03-24T11:17:08.584 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph-authtool: stdout AQC0CwBm0R7gABAAyhPcuosdHp7bUN5PMhxm+A== 2024-03-24T11:17:09.515 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph-authtool: stdout AQC0CwBmR8rpNhAAQjbcdSzk2rWIzirJFEy4YQ== 2024-03-24T11:17:10.462 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph-authtool: stdout AQC1CwBmqza4NBAAVnkU+VGraSbsLngnko0Dzw== 2024-03-24T11:17:10.463 INFO:teuthology.orchestra.run.smithi184.stdout:Creating initial monmap... 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool for a [v2:172.21.15.184:3300,v1:172.21.15.184:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run.smithi184.stdout:setting min_mon_release = quincy 2024-03-24T11:17:11.410 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: set fsid to f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:11.410 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:17:11.410 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:17:11.410 INFO:teuthology.orchestra.run.smithi184.stdout:Creating mon... 2024-03-24T11:17:12.348 INFO:teuthology.orchestra.run.smithi184.stdout:create mon.a on 2024-03-24T11:17:12.718 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:17:12.891 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715.target. 2024-03-24T11:17:12.891 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715.target. 2024-03-24T11:17:13.120 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:17:13.120 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T11:17:13.281 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715.target.wants/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@.service. 2024-03-24T11:17:13.620 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 podman[25292]: 2024-03-24 11:17:13.412116961 +0000 UTC m=+0.017815128 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:17:13.762 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:17:13.763 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-03-24T11:17:13.769 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:17:13.769 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-03-24T11:17:13.770 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-03-24T11:17:13.770 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:17:13.770 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mon to start... 2024-03-24T11:17:13.770 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mon... 2024-03-24T11:17:13.985 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 podman[25292]: 2024-03-24 11:17:13.691110745 +0000 UTC m=+0.296808908 container init 3bb4534b80e1784ddf5a3a3ce49262d5df6679db1d402f1413f1efe10ab564cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5) 2024-03-24T11:17:13.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 podman[25292]: 2024-03-24 11:17:13.696724439 +0000 UTC m=+0.302422611 container start 3bb4534b80e1784ddf5a3a3ce49262d5df6679db1d402f1413f1efe10ab564cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:17:13.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 bash[25292]: 3bb4534b80e1784ddf5a3a3ce49262d5df6679db1d402f1413f1efe10ab564cb 2024-03-24T11:17:13.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 systemd[1]: Started Ceph mon.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:17:13.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 ceph-mon[25330]: mkfs f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:13.986 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:13 smithi184 ceph-mon[25330]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:17:14.871 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:14 smithi184 ceph-mon[25330]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:17:14.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:14 smithi184 ceph-mon[25330]: monmap e1: 1 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:17:14.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:14 smithi184 ceph-mon[25330]: fsmap 2024-03-24T11:17:14.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:14 smithi184 ceph-mon[25330]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:17:14.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:14 smithi184 ceph-mon[25330]: mgrmap e1: no daemons active 2024-03-24T11:17:14.872 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:14 smithi184 ceph-mon[25330]: from='client.? 172.21.15.184:0/2321771309' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:17:14.877 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T11:17:14.877 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout id: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:14.877 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T11:17:14.877 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:14.877 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout services: 2024-03-24T11:17:14.877 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.53089s) 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout data: 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:mon is available 2024-03-24T11:17:14.878 INFO:teuthology.orchestra.run.smithi184.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T11:17:15.881 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:15.881 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:17:15.881 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout fsid = f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.184:3300,v1:172.21.15.184:6789] 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:17:15.882 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:17:15.883 INFO:teuthology.orchestra.run.smithi184.stdout:Generating new minimal ceph.conf... 2024-03-24T11:17:16.086 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:15 smithi184 ceph-mon[25330]: from='client.? 172.21.15.184:0/2297418562' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:17:16.086 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:15 smithi184 ceph-mon[25330]: from='client.? 172.21.15.184:0/2297418562' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:17:16.837 INFO:teuthology.orchestra.run.smithi184.stdout:Restarting the monitor... 2024-03-24T11:17:17.082 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:16 smithi184 ceph-mon[25330]: from='client.? 172.21.15.184:0/814476959' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:17.082 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:16 smithi184 systemd[1]: Stopping Ceph mon.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:17:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a[25307]: 2024-03-24T11:17:17.080+0000 7f2665760700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:17:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a[25307]: 2024-03-24T11:17:17.080+0000 7f2665760700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:17:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 podman[25660]: 2024-03-24 11:17:17.234949818 +0000 UTC m=+0.239412263 container died 3bb4534b80e1784ddf5a3a3ce49262d5df6679db1d402f1413f1efe10ab564cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:17:17.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 podman[25660]: 2024-03-24 11:17:17.553031879 +0000 UTC m=+0.557494392 container cleanup 3bb4534b80e1784ddf5a3a3ce49262d5df6679db1d402f1413f1efe10ab564cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T11:17:17.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 bash[25660]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:17:17.898 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 podman[25680]: 2024-03-24 11:17:17.695429562 +0000 UTC m=+0.451515154 container remove 3bb4534b80e1784ddf5a3a3ce49262d5df6679db1d402f1413f1efe10ab564cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T11:17:18.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T11:17:18.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 systemd[1]: Stopped Ceph mon.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:17:18.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:17 smithi184 systemd[1]: Starting Ceph mon.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:17:18.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 podman[25778]: 2024-03-24 11:17:18.121154329 +0000 UTC m=+0.113425493 container create b5a59501fd4016e04b7d359b2a19f05f0121106fb0b6e53b0fb079e06bd0db07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T11:17:18.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 podman[25778]: 2024-03-24 11:17:18.025373631 +0000 UTC m=+0.017644759 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:17:18.426 INFO:teuthology.orchestra.run.smithi184.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T11:17:18.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 podman[25778]: 2024-03-24 11:17:18.306749217 +0000 UTC m=+0.299020357 container init b5a59501fd4016e04b7d359b2a19f05f0121106fb0b6e53b0fb079e06bd0db07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-24T11:17:18.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 podman[25778]: 2024-03-24 11:17:18.311863801 +0000 UTC m=+0.304134921 container start b5a59501fd4016e04b7d359b2a19f05f0121106fb0b6e53b0fb079e06bd0db07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:17:18.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: pidfile_write: ignore empty --pid-file 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: load: jerasure load: lrc 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Git sha 0 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: DB SUMMARY 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: DB Session ID: O2ZDN96GDDGR8AR9HSUZ 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:17:18.635 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84653 ; 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.env: 0x55c9125edae0 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.info_log: 0x55c912e7d8c0 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:17:18.636 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.statistics: (nil) 2024-03-24T11:17:18.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.use_fsync: 0 2024-03-24T11:17:18.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:17:18.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:17:18.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:17:18.637 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.db_log_dir: 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.wal_dir: 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:17:18.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.write_buffer_manager: 0x55c912e51400 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:17:18.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.unordered_write: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.row_cache: None 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.wal_filter: None 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:17:18.640 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.wal_compression: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:17:18.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_open_files: -1 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:17:18.642 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Compression algorithms supported: 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kZSTD supported: 0 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:17:18.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.merge_operator: 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_filter: None 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c912e7d9c0) 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_type: 0 2024-03-24T11:17:18.644 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_index_type: 0 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_shortening: 1 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: checksum: 4 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: no_block_cache: 0 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache: 0x55c912e81090 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_options: 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: capacity : 536870912 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_shard_bits : 4 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: strict_capacity_limit : 0 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_cache_compressed: (nil) 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: persistent_cache: (nil) 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size: 4096 2024-03-24T11:17:18.645 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_size_deviation: 10 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_restart_interval: 16 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: index_block_restart_interval: 1 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: metadata_block_size: 4096 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: partition_filters: 0 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: use_delta_encoding: 1 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: filter_policy: bloomfilter 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: whole_key_filtering: 1 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: verify_compression: 0 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: format_version: 5 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_index_compression: 1 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: block_align: 0 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: max_auto_readahead_size: 262144 2024-03-24T11:17:18.646 INFO:journalctl@ceph.mon.a.smithi184.stdout: prepopulate_block_cache: 0 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression: NoCompression 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.num_levels: 7 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:17:18.647 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:17:18.648 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:17:18.649 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:17:18.650 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:17:18.650 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:17:18.651 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:17:18.652 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:17:18.653 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.ttl: 2592000 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:17:18.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5f428f95-5ff3-4879-a82f-6be03a911d92 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279038341103, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279038341949, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81278, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79465, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73774, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711279038, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5f428f95-5ff3-4879-a82f-6be03a911d92", "db_session_id": "O2ZDN96GDDGR8AR9HSUZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:17:18.655 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279038342029, "job": 1, "event": "recovery_finished"} 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c912f72000 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: DB pointer 0x55c912f5c000 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** DB Stats ** 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:17:18.656 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: L0 2/0 81.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: Sum 2/0 81.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** Compaction Stats [default] ** 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-03-24T11:17:18.657 INFO:journalctl@ceph.mon.a.smithi184.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: Cumulative compaction: 0.00 GB write, 27.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: Interval compaction: 0.00 GB write, 27.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:17:18.658 INFO:journalctl@ceph.mon.a.smithi184.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache BinnedLRUCache@0x55c912e81090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: starting mon.a rank 0 at public addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] at bind addrs [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???) e1 preinit fsid f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).mds e1 new map 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).mds e1 print_map 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: e1 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: legacy client fscid: -1 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: 2024-03-24T11:17:18.659 INFO:journalctl@ceph.mon.a.smithi184.stdout: No filesystems configured 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: monmap e1: 1 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: fsmap 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:17:18.660 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 ceph-mon[25816]: mgrmap e1: no daemons active 2024-03-24T11:17:18.661 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 bash[25778]: b5a59501fd4016e04b7d359b2a19f05f0121106fb0b6e53b0fb079e06bd0db07 2024-03-24T11:17:18.661 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:18 smithi184 systemd[1]: Started Ceph mon.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:17:19.431 INFO:teuthology.orchestra.run.smithi184.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:17:19.453 INFO:teuthology.orchestra.run.smithi184.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:17:19.453 INFO:teuthology.orchestra.run.smithi184.stdout:Creating mgr... 2024-03-24T11:17:19.454 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T11:17:19.454 INFO:teuthology.orchestra.run.smithi184.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T11:17:19.696 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:17:19.696 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T11:17:19.850 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715.target.wants/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@.service. 2024-03-24T11:17:20.169 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:19 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3608861770' entity='client.admin' 2024-03-24T11:17:20.169 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 podman[26058]: 2024-03-24 11:17:19.993809725 +0000 UTC m=+0.018365461 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:17:20.355 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:17:20.355 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-03-24T11:17:20.362 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:17:20.362 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-03-24T11:17:20.362 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-03-24T11:17:20.362 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:17:20.372 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:17:20.372 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-03-24T11:17:20.380 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:17:20.380 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-03-24T11:17:20.381 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-03-24T11:17:20.381 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T11:17:20.381 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr to start... 2024-03-24T11:17:20.381 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr... 2024-03-24T11:17:20.492 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 podman[26058]: 2024-03-24 11:17:20.241396272 +0000 UTC m=+0.265952009 container init 85500b4469ea705d3f90bf9af058cf813cd4eebd2d3bdd2a17eae28f4c55abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:17:20.492 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 podman[26058]: 2024-03-24 11:17:20.246516934 +0000 UTC m=+0.271072666 container start 85500b4469ea705d3f90bf9af058cf813cd4eebd2d3bdd2a17eae28f4c55abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-24T11:17:20.493 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 bash[26058]: 85500b4469ea705d3f90bf9af058cf813cd4eebd2d3bdd2a17eae28f4c55abef 2024-03-24T11:17:20.493 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 systemd[1]: Started Ceph mgr.y for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:17:21.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:20 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3531097653' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:17:21.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:20.818+0000 7f4aa74ba200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:17:21.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:20.915+0000 7f4aa74ba200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:17:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:17:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsid": "f43a2692-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:17:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:17:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:17:21.528 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:17:21.529 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:17:21.530 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:17:21.533 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:17:21.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:17:13.767960+0000", 2024-03-24T11:17:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:17:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:17:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:17:21.536 INFO:teuthology.orchestra.run.smithi184.stdout:mgr not available, waiting (1/15)... 2024-03-24T11:17:21.816 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:21.580+0000 7f4aa74ba200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:17:21.816 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:21.674+0000 7f4aa74ba200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:17:21.816 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:21.815+0000 7f4aa74ba200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:17:22.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:21.910+0000 7f4aa74ba200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:17:22.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:21.990+0000 7f4aa74ba200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:17:22.533 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:22 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:22.250+0000 7f4aa74ba200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:17:22.533 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:22 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:22.356+0000 7f4aa74ba200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:17:22.533 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:22 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:22.440+0000 7f4aa74ba200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:17:22.533 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:22 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:22.532+0000 7f4aa74ba200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:17:22.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:22 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:22.707+0000 7f4aa74ba200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:17:23.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:23 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:23.674+0000 7f4aa74ba200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:17:23.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:23 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:23.809+0000 7f4aa74ba200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:17:24.361 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:24 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2528323360' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:17:24.361 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:24.155+0000 7f4aa74ba200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:17:24.361 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:24.244+0000 7f4aa74ba200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:17:24.620 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:24.440+0000 7f4aa74ba200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:17:24.628 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsid": "f43a2692-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:17:24.629 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:17:24.630 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:17:24.631 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:24.632 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:17:13.767960+0000", 2024-03-24T11:17:24.633 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:17:24.633 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:24.633 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:17:24.633 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:17:24.633 INFO:teuthology.orchestra.run.smithi184.stdout:mgr not available, waiting (2/15)... 2024-03-24T11:17:24.898 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:24.621+0000 7f4aa74ba200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:17:25.157 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:24.898+0000 7f4aa74ba200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:17:25.158 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:25 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:25.157+0000 7f4aa74ba200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:17:25.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:25 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:25.386+0000 7f4aa74ba200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:17:26.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:26 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:26.180+0000 7f4aa74ba200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:17:27.086 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:26 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:26.770+0000 7f4aa74ba200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:17:27.086 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:26 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:26.869+0000 7f4aa74ba200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:17:27.086 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:27.085+0000 7f4aa74ba200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:17:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: Activating manager daemon y 2024-03-24T11:17:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: mgrmap e2: y(active, starting, since 0.00303591s) 2024-03-24T11:17:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:17:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:17:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: Manager daemon y is now available 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' 2024-03-24T11:17:27.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:27 smithi184 ceph-mon[25816]: from='mgr.14100 172.21.15.184:0/2644098246' entity='mgr.y' 2024-03-24T11:17:29.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:29 smithi184 ceph-mon[25816]: mgrmap e3: y(active, since 1.00685s) 2024-03-24T11:17:30.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:30 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/617063838' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:17:31.376 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:31 smithi184 ceph-mon[25816]: mgrmap e4: y(active, since 3s) 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsid": "f43a2692-e9cf-11ee-95cd-87774f69a715", 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:17:34.635 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:17:34.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:17:34.637 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:17:34.638 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:17:13.767960+0000", 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:17:34.639 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:17:34.640 INFO:teuthology.orchestra.run.smithi184.stdout:mgr is available 2024-03-24T11:17:35.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:35 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/4026536349' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:17:35.604 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:35 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/4026536349' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:17:35.858 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:35.858 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout fsid = f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:17:35.859 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling cephadm module... 2024-03-24T11:17:36.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:36 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3352328701' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:17:36.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:36 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ignoring --setuser ceph since I am not root 2024-03-24T11:17:36.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:36 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ignoring --setgroup ceph since I am not root 2024-03-24T11:17:37.537 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:37 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:37.189+0000 7f962bcda200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:17:37.538 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:37 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:37.287+0000 7f962bcda200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:17:37.905 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:37 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3352328701' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:17:37.906 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:37 smithi184 ceph-mon[25816]: mgrmap e5: y(active, since 9s) 2024-03-24T11:17:38.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:37 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:37.952+0000 7f962bcda200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:17:38.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.046+0000 7f962bcda200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:17:38.482 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.193+0000 7f962bcda200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:17:38.482 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.289+0000 7f962bcda200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:17:38.482 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.372+0000 7f962bcda200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:17:38.486 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:17:38.486 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T11:17:38.487 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:17:38.487 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:17:38.487 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:17:38.487 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:17:38.487 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for the mgr to restart... 2024-03-24T11:17:38.487 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr epoch 5... 2024-03-24T11:17:38.760 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/4101812733' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:17:38.760 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.644+0000 7f962bcda200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:17:38.760 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.758+0000 7f962bcda200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:17:39.124 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.845+0000 7f962bcda200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:17:39.124 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:38 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:38.943+0000 7f962bcda200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:17:39.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:39 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:39.122+0000 7f962bcda200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:17:40.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:40 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:40.076+0000 7f962bcda200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:17:40.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:40 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:40.205+0000 7f962bcda200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:17:40.826 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:40 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:40.541+0000 7f962bcda200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:17:40.826 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:40 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:40.631+0000 7f962bcda200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:17:40.826 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:40 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:40.824+0000 7f962bcda200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:17:41.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:41.006+0000 7f962bcda200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:17:41.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:41.288+0000 7f962bcda200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:17:41.550 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:41.548+0000 7f962bcda200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:17:41.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:41.785+0000 7f962bcda200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:17:42.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:42 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:42.584+0000 7f962bcda200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:17:43.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:43.156+0000 7f962bcda200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:17:43.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:43.252+0000 7f962bcda200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:17:43.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:17:43.467+0000 7f962bcda200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:17:43.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: Active manager daemon y restarted 2024-03-24T11:17:43.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: Activating manager daemon y 2024-03-24T11:17:43.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: mgrmap e6: y(active, starting, since 0.00355252s) 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: Manager daemon y is now available 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:43.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:43 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:44.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: Found migration_current of "None". Setting to last migration. 2024-03-24T11:17:44.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:44.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:17:44.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:17:44.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:44.864 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:44.865 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:44 smithi184 ceph-mon[25816]: mgrmap e7: y(active, since 1.00544s) 2024-03-24T11:17:45.141 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:17:45.141 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T11:17:45.141 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:17:45.141 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:17:45.142 INFO:teuthology.orchestra.run.smithi184.stdout:mgr epoch 5 is available 2024-03-24T11:17:45.142 INFO:teuthology.orchestra.run.smithi184.stdout:Setting orchestrator backend to cephadm... 2024-03-24T11:17:45.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:45 smithi184 ceph-mon[25816]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:17:45.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:45 smithi184 ceph-mon[25816]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:17:45.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:45 smithi184 ceph-mon[25816]: [24/Mar/2024:11:17:44] ENGINE Bus STARTING 2024-03-24T11:17:45.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:45 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:46.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:46 smithi184 ceph-mon[25816]: [24/Mar/2024:11:17:44] ENGINE Serving on http://172.21.15.184:8765 2024-03-24T11:17:46.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:46 smithi184 ceph-mon[25816]: [24/Mar/2024:11:17:44] ENGINE Serving on https://172.21.15.184:7150 2024-03-24T11:17:46.772 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:46 smithi184 ceph-mon[25816]: [24/Mar/2024:11:17:44] ENGINE Bus STARTED 2024-03-24T11:17:46.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:46 smithi184 ceph-mon[25816]: mgrmap e8: y(active, since 2s) 2024-03-24T11:17:46.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:46 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:46.773 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:46 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:47.634 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:47 smithi184 ceph-mon[25816]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:47.636 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T11:17:47.637 INFO:teuthology.orchestra.run.smithi184.stdout:Generating ssh key... 2024-03-24T11:17:48.831 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-mon[25816]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: Generating public/private rsa key pair. 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: Your identification has been saved in /tmp/tmp995tw37i/key. 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: Your public key has been saved in /tmp/tmp995tw37i/key.pub. 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: The key fingerprint is: 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: SHA256:vSFtrqjnNAFaOUrwpU4FctZpimwEd/nSjHZpbV95OEI ceph-f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: The key's randomart image is: 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: +---[RSA 3072]----+ 2024-03-24T11:17:49.165 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: |= =o+o | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: | O ==. E | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: |o.=o== o . o | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: |.*.++oB oo. = . | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: |. +. +..S.=o o | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: | . +.o | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: | o o | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: | ..o . | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: | .+o . | 2024-03-24T11:17:49.166 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:17:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: +----[SHA256]-----+ 2024-03-24T11:17:49.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:49 smithi184 ceph-mon[25816]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:49.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:49 smithi184 ceph-mon[25816]: Generating ssh key... 2024-03-24T11:17:49.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:49 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:49.759 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:49 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:50.656 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:17:50.657 INFO:teuthology.orchestra.run.smithi184.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:17:50.657 INFO:teuthology.orchestra.run.smithi184.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T11:17:50.657 INFO:teuthology.orchestra.run.smithi184.stdout:Adding host smithi184... 2024-03-24T11:17:50.904 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:50 smithi184 ceph-mon[25816]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:51.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:51 smithi184 ceph-mon[25816]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "addr": "172.21.15.184", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:53.174 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:52 smithi184 ceph-mon[25816]: Deploying cephadm binary to smithi184 2024-03-24T11:17:56.090 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Added host 'smithi184' with addr '172.21.15.184' 2024-03-24T11:17:56.090 INFO:teuthology.orchestra.run.smithi184.stdout:Deploying unmanaged mon service... 2024-03-24T11:17:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:56 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:56 smithi184 ceph-mon[25816]: Added host smithi184 2024-03-24T11:17:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:56 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:17:57.540 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T11:17:57.540 INFO:teuthology.orchestra.run.smithi184.stdout:Deploying unmanaged mgr service... 2024-03-24T11:17:58.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:57 smithi184 ceph-mon[25816]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:58.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:57 smithi184 ceph-mon[25816]: Saving service mon spec with placement count:5 2024-03-24T11:17:58.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:57 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:59.424 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T11:17:59.742 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:59 smithi184 ceph-mon[25816]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:59.742 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:59 smithi184 ceph-mon[25816]: Saving service mgr spec with placement count:2 2024-03-24T11:17:59.742 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:59 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:17:59.742 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:17:59 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:18:00.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:00 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:18:00.930 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:00 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2610009827' entity='client.admin' 2024-03-24T11:18:02.333 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling the dashboard module... 2024-03-24T11:18:02.666 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:02 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3372325889' entity='client.admin' 2024-03-24T11:18:03.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ignoring --setuser ceph since I am not root 2024-03-24T11:18:03.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ignoring --setgroup ceph since I am not root 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='mgr.14118 172.21.15.184:0/1783505137' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]': finished 2024-03-24T11:18:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:03 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3386605935' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:18:04.280 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:04 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:04.021+0000 7f6cc1a96200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:18:04.280 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:04 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:04.120+0000 7f6cc1a96200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:18:04.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:04 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3386605935' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:18:04.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:04 smithi184 ceph-mon[25816]: mgrmap e9: y(active, since 19s) 2024-03-24T11:18:05.025 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:04 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:04.786+0000 7f6cc1a96200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:18:05.025 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:04 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:04.880+0000 7f6cc1a96200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:18:05.025 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.024+0000 7f6cc1a96200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:18:05.308 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.121+0000 7f6cc1a96200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:18:05.308 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.204+0000 7f6cc1a96200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:18:05.311 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:18:05.311 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T11:18:05.311 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:18:05.312 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T11:18:05.312 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:18:05.312 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:18:05.312 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for the mgr to restart... 2024-03-24T11:18:05.312 INFO:teuthology.orchestra.run.smithi184.stdout:Waiting for mgr epoch 9... 2024-03-24T11:18:05.565 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1002392283' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:18:05.565 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.478+0000 7f6cc1a96200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:18:05.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.594+0000 7f6cc1a96200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:18:05.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.682+0000 7f6cc1a96200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:18:05.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.782+0000 7f6cc1a96200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:18:06.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:05 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:05.959+0000 7f6cc1a96200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:18:07.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:06.904+0000 7f6cc1a96200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:18:07.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:07 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:07.032+0000 7f6cc1a96200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:18:07.644 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:07 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:07.363+0000 7f6cc1a96200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:18:07.645 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:07 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:07.451+0000 7f6cc1a96200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:18:07.645 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:07 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:07.644+0000 7f6cc1a96200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:18:07.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:07 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:07.826+0000 7f6cc1a96200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:18:08.366 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:08 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:08.106+0000 7f6cc1a96200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:18:08.366 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:08 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:08.365+0000 7f6cc1a96200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:18:08.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:08 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:08.598+0000 7f6cc1a96200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:18:09.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:09 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:09.386+0000 7f6cc1a96200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:18:10.263 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:09 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:09.953+0000 7f6cc1a96200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:18:10.264 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:10.049+0000 7f6cc1a96200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:18:10.264 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:18:10.263+0000 7f6cc1a96200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:18:10.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: Active manager daemon y restarted 2024-03-24T11:18:10.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: Activating manager daemon y 2024-03-24T11:18:10.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:18:10.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: mgrmap e10: y(active, starting, since 0.00325575s) 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: Manager daemon y is now available 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:10.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:11.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:18:11.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:18:11.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:11 smithi184 ceph-mon[25816]: [24/Mar/2024:11:18:10] ENGINE Bus STARTING 2024-03-24T11:18:11.599 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:11 smithi184 ceph-mon[25816]: mgrmap e11: y(active, since 1.00583s) 2024-03-24T11:18:12.153 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:18:12.153 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T11:18:12.153 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:18:12.153 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:18:12.154 INFO:teuthology.orchestra.run.smithi184.stdout:mgr epoch 9 is available 2024-03-24T11:18:12.154 INFO:teuthology.orchestra.run.smithi184.stdout:Generating a dashboard self-signed certificate... 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: [24/Mar/2024:11:18:10] ENGINE Serving on http://172.21.15.184:8765 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: [24/Mar/2024:11:18:10] ENGINE Serving on https://172.21.15.184:7150 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: [24/Mar/2024:11:18:10] ENGINE Bus STARTED 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:12.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:12.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:12.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:12 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-03-24T11:18:13.535 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T11:18:13.535 INFO:teuthology.orchestra.run.smithi184.stdout:Creating initial admin user... 2024-03-24T11:18:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:13 smithi184 ceph-mon[25816]: Deploying daemon agent.smithi184 on smithi184 2024-03-24T11:18:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:13 smithi184 ceph-mon[25816]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:13 smithi184 ceph-mon[25816]: mgrmap e12: y(active, since 2s) 2024-03-24T11:18:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:14.939 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$7TBvZ8UxbnW711BLBwLmV.tdHTZcRSCcrwaaVhZkB8AhqkVbtqIJi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711279094, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:18:14.939 INFO:teuthology.orchestra.run.smithi184.stdout:Fetching dashboard port number... 2024-03-24T11:18:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:15 smithi184 ceph-mon[25816]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:15 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:16.146 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T11:18:16.155 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:18:16.156 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout disabled 2024-03-24T11:18:16.162 INFO:teuthology.orchestra.run.smithi184.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:18:16.162 INFO:teuthology.orchestra.run.smithi184.stdout:systemctl: stdout inactive 2024-03-24T11:18:16.162 INFO:teuthology.orchestra.run.smithi184.stdout:firewalld.service is not enabled 2024-03-24T11:18:16.162 INFO:teuthology.orchestra.run.smithi184.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:18:16.164 INFO:teuthology.orchestra.run.smithi184.stdout:Ceph Dashboard is now available at: 2024-03-24T11:18:16.164 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: URL: https://smithi184.front.sepia.ceph.com:8443/ 2024-03-24T11:18:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: User: admin 2024-03-24T11:18:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: Password: 59gy09fug3 2024-03-24T11:18:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:16.165 INFO:teuthology.orchestra.run.smithi184.stdout:Saving cluster configuration to /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config directory 2024-03-24T11:18:16.166 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling autotune for osd_memory_target 2024-03-24T11:18:16.570 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:16 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1743710299' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:18:17.432 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:17 smithi184 ceph-mon[25816]: mgrmap e13: y(active, since 6s) 2024-03-24T11:18:18.682 INFO:teuthology.orchestra.run.smithi184.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T11:18:18.682 INFO:teuthology.orchestra.run.smithi184.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T11:18:18.682 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.682 INFO:teuthology.orchestra.run.smithi184.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.683 INFO:teuthology.orchestra.run.smithi184.stdout: ceph telemetry on 2024-03-24T11:18:18.684 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.684 INFO:teuthology.orchestra.run.smithi184.stdout:For more information see: 2024-03-24T11:18:18.684 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.684 INFO:teuthology.orchestra.run.smithi184.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T11:18:18.684 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:18.684 INFO:teuthology.orchestra.run.smithi184.stdout:Bootstrap complete. 2024-03-24T11:18:18.726 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:18:18.726 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:18:18.726 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:18:18.741 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:18:18.741 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:18:18.741 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:18:18.796 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:18:18.796 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:18:18.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T11:18:18.946 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:18:18.946 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:18:18.946 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:18:19.002 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:18:19.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:18:19.151 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:18:19.166 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:18:19.213 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:18:19.228 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:18:19.275 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTJkt9hZdU1cwCjZiodFIEoZricgXBfqFz5eYVSdaZvAcYsgVZD3CW8hK90WLuJPJK6AO9Puvw2MeW1QSqaMYgBVibpNU6xjiusjCH8hCTl0WZKUImQXsCWigvP6uIedQ/Ev9QxeO4XqlxORU3hjvumqVaGvpoBJX5FLOO8wjhgepXFRxw40Vuqg/3cS1h/74+BCqY3HsOlUr/rlwsSLAB+a14LoWKpxStVClSWievzyzl2JAt37NkFgQCsS1BP+maXq/VqV6tV7GkMBzKUMwUADANfA7eHF6dq3IeORiKK15N4vNsEhlSfSaTaqjJC3+1sdX/lDClnuivO8DNxMcXMJrGjq60nrNQLc4T1yyHjTAyW+XzaWcdquLUuVcAph4Qn0tOadA8+D4FiAtheJ/2oJoFu8RdLK8HJJIE3vAasOZI/Z5LJGBDGBFxO9JjR+0MHOMvgd1LXkkByG0RO8kPUbVJVaRLkKOoliaVWEisRWShZM9vfAdFiHWdr9fc3m8= ceph-f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:18:19.290 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:18:19.317 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:19 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1727750989' entity='client.admin' 2024-03-24T11:18:19.678 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:21.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:21.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.050 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:21.051 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:20 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:22.322 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:18:22.322 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:18:22.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:22 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/198920217' entity='client.admin' 2024-03-24T11:18:22.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:22.625 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:23.355 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:26.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:26.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:26.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:26.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:26.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:26.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:26.249 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-03-24T11:18:26.250 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:18:26.250 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:18:26.269 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:18:26.269 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:26.327 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-03-24T11:18:26.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch host add smithi186 2024-03-24T11:18:26.753 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:27.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:27.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:28.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:28 smithi184 ceph-mon[25816]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:29.822 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:29 smithi184 ceph-mon[25816]: Deploying cephadm binary to smithi186 2024-03-24T11:18:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:31 smithi184 ceph-mon[25816]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:32.872 INFO:teuthology.orchestra.run.smithi184.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-03-24T11:18:33.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:33 smithi184 ceph-mon[25816]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:33.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:33 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:33.460 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:33 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:33.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:18:33.735 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:34.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: Added host smithi186 2024-03-24T11:18:34.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:34.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:34.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:34.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:34.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:34.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:34 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:35.451 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:35.451 INFO:teuthology.orchestra.run.smithi184.stdout:[{"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]: dispatch 2024-03-24T11:18:35.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:35 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]': finished 2024-03-24T11:18:36.073 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-03-24T11:18:36.073 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:18:36.073 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:18:36.092 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:18:36.092 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:36.149 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-03-24T11:18:36.149 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch host add smithi189 2024-03-24T11:18:36.384 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:36.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:36 smithi184 ceph-mon[25816]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:18:36.641 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:36 smithi184 ceph-mon[25816]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: Deploying daemon agent.smithi186 on smithi186 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:37.638 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:37.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:37.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:37.639 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:39.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:39 smithi184 ceph-mon[25816]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:39.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:39 smithi184 ceph-mon[25816]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:40.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:40 smithi184 ceph-mon[25816]: Deploying cephadm binary to smithi189 2024-03-24T11:18:40.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:40.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:40.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:41.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:41 smithi184 ceph-mon[25816]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:42.460 INFO:teuthology.orchestra.run.smithi184.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-03-24T11:18:42.715 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:42.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:42.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:42.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:42.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:42.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: Added host smithi189 2024-03-24T11:18:42.716 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:43.070 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:18:43.831 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:44.066 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:44.067 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:44.067 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:44.067 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:44.067 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:44.067 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:44.067 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:43 smithi184 ceph-mon[25816]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:18:45.519 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:18:45.519 INFO:teuthology.orchestra.run.smithi184.stdout:[{"addr": "172.21.15.184", "hostname": "smithi184", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:18:45.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]': finished 2024-03-24T11:18:46.188 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:18:46.188 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:18:46.427 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: Deploying daemon agent.smithi189 on smithi189 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:46.549 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:46.550 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:47.707 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:47 smithi184 ceph-mon[25816]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:18:47.708 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:47 smithi184 ceph-mon[25816]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:48.458 INFO:teuthology.orchestra.run.smithi184.stderr:adjusted tunables profile to default 2024-03-24T11:18:48.739 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:48 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2539094880' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:18:49.018 INFO:tasks.cephadm:Adding mon.a on smithi184 2024-03-24T11:18:49.018 INFO:tasks.cephadm:Adding mon.b on smithi186 2024-03-24T11:18:49.018 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-03-24T11:18:49.018 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c' 2024-03-24T11:18:49.246 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:49.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:49 smithi184 ceph-mon[25816]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:49.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:49 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2539094880' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:18:49.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:49 smithi184 ceph-mon[25816]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:18:49.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:49 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:49.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:49 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:50.909 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-03-24T11:18:51.415 DEBUG:teuthology.orchestra.run.smithi186:mon.b> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T11:18:51.418 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T11:18:51.421 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:18:51.421 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:18:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:51.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:51.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:18:51.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:52.160 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:52.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:52 smithi184 ceph-mon[25816]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:52.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:52 smithi184 ceph-mon[25816]: Saving service mon spec with placement smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:18:52.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:52 smithi184 ceph-mon[25816]: Deploying daemon mon.c on smithi189 2024-03-24T11:18:53.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:53 smithi184 ceph-mon[25816]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:54.761 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:18:54.761 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:17:10.805235Z","created":"2024-03-24T11:17:10.805235Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:18:54.762 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-03-24T11:18:55.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:55 smithi184 ceph-mon[25816]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:55.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:18:55 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/798013294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:18:56.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:18:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:18:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:18:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Added host smithi186 2024-03-24T11:18:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.420 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:18:56.420 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:18:56.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:56.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]: dispatch 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]': finished 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Deploying daemon agent.smithi186 on smithi186 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.618 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Deploying cephadm binary to smithi189 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Added host smithi189 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.620 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]': finished 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Deploying daemon agent.smithi189 on smithi189 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.621 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.622 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2539094880' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2539094880' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.623 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Saving service mon spec with placement smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: Deploying daemon mon.c on smithi189 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/798013294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 ceph-mon[27694]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 bash[27656]: 9c795b036f71bbb522ae433227ec02e7a7e614cfffaf79b8ad296931caa7e59a 2024-03-24T11:18:56.624 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:18:56 smithi189 systemd[1]: Started Ceph mon.c for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:18:56.654 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:19:01.643 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: mon.a calling monitor election 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: mon.c calling monitor election 2024-03-24T11:19:01.644 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: monmap e2: 2 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: fsmap 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: mgrmap e13: y(active, since 51s) 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: overall HEALTH_OK 2024-03-24T11:19:01.645 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:01.646 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:01 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:01.732 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:19:01.733 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:18:56.303809Z","created":"2024-03-24T11:17:10.805235Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:19:01.733 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:01.733 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:01.733 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: mon.a calling monitor election 2024-03-24T11:19:01.733 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.733 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.733 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: mon.c calling monitor election 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: monmap e2: 2 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: fsmap 2024-03-24T11:19:01.734 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: mgrmap e13: y(active, since 51s) 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: overall HEALTH_OK 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:01.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:01 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:01.736 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-03-24T11:19:02.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:02 smithi189 ceph-mon[27694]: Deploying daemon mon.b on smithi186 2024-03-24T11:19:02.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:02 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:02.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:02 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:02.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:02 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:02.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:02 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/2014010678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:02.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:02 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:02.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:02 smithi184 ceph-mon[25816]: Deploying daemon mon.b on smithi186 2024-03-24T11:19:02.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:02 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:02.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:02 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:02.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:02 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:02.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:02 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/2014010678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:02.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:02 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:03.309 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:19:03.309 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:19:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:19:03.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:03 smithi189 ceph-mon[27694]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:03.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:03 smithi184 ceph-mon[25816]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.395 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:19:05.396 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:18:56.303809Z","created":"2024-03-24T11:17:10.805235Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:19:05.397 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-03-24T11:19:05.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 podman[27561]: 2024-03-24 11:19:05.255553027 +0000 UTC m=+0.017985487 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:19:05.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:05 smithi184 ceph-mon[25816]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:05 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/3050999265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:05 smithi189 ceph-mon[27694]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.692 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:05 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/3050999265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 podman[27561]: 2024-03-24 11:19:05.539692861 +0000 UTC m=+0.302125327 container init f2cf77d271fdc846048210e3fc3c477f30e6a65d75a37df00793ce03a9a8531c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 podman[27561]: 2024-03-24 11:19:05.545226617 +0000 UTC m=+0.307659073 container start f2cf77d271fdc846048210e3fc3c477f30e6a65d75a37df00793ce03a9a8531c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pidfile_write: ignore empty --pid-file 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: load: jerasure load: lrc 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: RocksDB version: 7.9.2 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Git sha 0 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: DB SUMMARY 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: DB Session ID: 79Q7XB97HIZ3QWSKSP82 2024-03-24T11:19:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T11:19:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.env: 0x55ff2f627ae0 2024-03-24T11:19:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T11:19:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.info_log: 0x55ff3192b8c0 2024-03-24T11:19:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:19:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.statistics: (nil) 2024-03-24T11:19:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.use_fsync: 0 2024-03-24T11:19:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:19:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.db_log_dir: 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.wal_dir: 2024-03-24T11:19:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:19:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.write_buffer_manager: 0x55ff318ff400 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.unordered_write: 0 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:19:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.row_cache: None 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.wal_filter: None 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.wal_compression: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:19:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:19:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_open_files: -1 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Compression algorithms supported: 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kZSTD supported: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:19:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.merge_operator: 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_filter: None 2024-03-24T11:19:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ff3192b9c0) 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_type: 0 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_index_type: 0 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_shortening: 1 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:19:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: checksum: 4 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: no_block_cache: 0 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache: 0x55ff3192f090 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_options: 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: capacity : 536870912 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_shard_bits : 4 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: strict_capacity_limit : 0 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_compressed: (nil) 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: persistent_cache: (nil) 2024-03-24T11:19:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size: 4096 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size_deviation: 10 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_restart_interval: 16 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_block_restart_interval: 1 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: metadata_block_size: 4096 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: partition_filters: 0 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: use_delta_encoding: 1 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: filter_policy: bloomfilter 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: whole_key_filtering: 1 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: verify_compression: 0 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: format_version: 5 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_index_compression: 1 2024-03-24T11:19:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_align: 0 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout: max_auto_readahead_size: 262144 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout: prepopulate_block_cache: 0 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout: initial_auto_readahead_size: 8192 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression: NoCompression 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.num_levels: 7 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:19:05.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:19:05.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:19:05.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:19:05.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:19:05.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.ttl: 2592000 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T11:19:05.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T11:19:05.814 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:19:05.814 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:19:05.814 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:19:05.814 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:19:05.814 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:19:05.814 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a9dcbd0f-fe14-409b-aaeb-3e0e4ec38346 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279145575191, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279145575602, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711279145, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a9dcbd0f-fe14-409b-aaeb-3e0e4ec38346", "db_session_id": "79Q7XB97HIZ3QWSKSP82", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711279145575680, "job": 1, "event": "recovery_finished"} 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:19:05.815 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ff31a20000 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: DB pointer 0x55ff31a0a000 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** DB Stats ** 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:19:05.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T11:19:05.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:19:05.818 INFO:journalctl@ceph.mon.b.smithi186.stdout: Block cache BinnedLRUCache@0x55ff3192f090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: using public_addr v2:172.21.15.186:0/0 -> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: starting mon.b rank -1 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(???) e0 preinit fsid f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: e1 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: legacy client fscid: -1 2024-03-24T11:19:05.819 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout: No filesystems configured 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Added host smithi186 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.820 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]: dispatch 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]': finished 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:19:05.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Deploying daemon agent.smithi186 on smithi186 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.822 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Deploying cephadm binary to smithi189 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Added host smithi189 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]': finished 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Deploying daemon agent.smithi189 on smithi189 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.825 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2539094880' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2539094880' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:19:05.826 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Saving service mon spec with placement smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Deploying daemon mon.c on smithi189 2024-03-24T11:19:05.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/798013294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.a calling monitor election 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.c calling monitor election 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.828 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: monmap e2: 2 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: fsmap 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mgrmap e13: y(active, since 51s) 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: overall HEALTH_OK 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:05.829 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: Deploying daemon mon.b on smithi186 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/2014010678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/3050999265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 ceph-mon[27599]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 bash[27561]: f2cf77d271fdc846048210e3fc3c477f30e6a65d75a37df00793ce03a9a8531c 2024-03-24T11:19:05.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:05 smithi186 systemd[1]: Started Ceph mon.b for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:19:06.969 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T11:19:06.969 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:19:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:19:10.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:10.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: mon.a calling monitor election 2024-03-24T11:19:10.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: mon.c calling monitor election 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:19:10.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: fsmap 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: mgrmap e13: y(active, since 60s) 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: mon.b (rank 2) addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum) 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:10.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: mon.a calling monitor election 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: mon.c calling monitor election 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: fsmap 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: mgrmap e13: y(active, since 60s) 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: mon.b (rank 2) addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum) 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.148 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T11:19:11.148 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","modified":"2024-03-24T11:19:05.598799Z","created":"2024-03-24T11:17:10.805235Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:3300","nonce":0},{"type":"v1","addr":"172.21.15.184:6789","nonce":0}]},"addr":"172.21.15.184:6789/0","public_addr":"172.21.15.184:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:19:11.150 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-03-24T11:19:11.782 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:19:11.783 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:11.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/1303401797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:11.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:11.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:11.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:11.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:11.868 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:11 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:19:12.098 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/1303401797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:12.099 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:12.100 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:11 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.694 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: mon.a calling monitor election 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: mon.c calling monitor election 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: fsmap 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: mgrmap e13: y(active, since 60s) 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: mon.b (rank 2) addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum) 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:19:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/1303401797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:12.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:12 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:13.536 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: mon.b calling monitor election 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: mon.b calling monitor election 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: mon.c calling monitor election 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: mon.a calling monitor election 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: fsmap 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: mgrmap e13: y(active, since 62s) 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: Cluster is now healthy 2024-03-24T11:19:13.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:13 smithi186 ceph-mon[27599]: overall HEALTH_OK 2024-03-24T11:19:13.612 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: mon.b calling monitor election 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: mon.b calling monitor election 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: mon.c calling monitor election 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: mon.a calling monitor election 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: fsmap 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: mgrmap e13: y(active, since 62s) 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: Cluster is now healthy 2024-03-24T11:19:13.613 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:13 smithi184 ceph-mon[25816]: overall HEALTH_OK 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: mon.b calling monitor election 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: mon.b calling monitor election 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: mon.c calling monitor election 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: mon.a calling monitor election 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: monmap e3: 3 mons at {a=[v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: fsmap 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: mgrmap e13: y(active, since 62s) 2024-03-24T11:19:13.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T11:19:13.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: Cluster is now healthy 2024-03-24T11:19:13.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:13 smithi189 ceph-mon[27694]: overall HEALTH_OK 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:14.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.751 INFO:teuthology.orchestra.run.smithi184.stdout:# minimal ceph.conf for f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:19:14.751 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2024-03-24T11:19:14.751 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:19:14.751 INFO:teuthology.orchestra.run.smithi184.stdout: mon_host = [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:14.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:14.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:14.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:14.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:15.398 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:19:15.399 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:19:15.399 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:19:15.428 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:19:15.428 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:19:15.495 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:19:15.496 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:19:15.527 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:19:15.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:19:15.597 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:19:15.597 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:19:15.629 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:15 smithi189 ceph-mon[27694]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:19:15.630 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:15 smithi189 ceph-mon[27694]: Reconfiguring daemon mon.b on smithi186 2024-03-24T11:19:15.630 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:15 smithi189 ceph-mon[27694]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:15.630 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:15 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2522431722' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:15.630 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:15 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:15.635 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:19:15.635 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:19:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:15 smithi184 ceph-mon[25816]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:19:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:15 smithi184 ceph-mon[25816]: Reconfiguring daemon mon.b on smithi186 2024-03-24T11:19:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:15 smithi184 ceph-mon[25816]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:15 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2522431722' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:15.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:15 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:15.704 INFO:tasks.cephadm:Adding mgr.y on smithi184 2024-03-24T11:19:15.704 INFO:tasks.cephadm:Adding mgr.x on smithi186 2024-03-24T11:19:15.704 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-03-24T11:19:15.704 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi184=y;smithi186=x;smithi189=z' 2024-03-24T11:19:15.791 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:15 smithi186 ceph-mon[27599]: Reconfiguring mon.b (monmap changed)... 2024-03-24T11:19:15.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:15 smithi186 ceph-mon[27599]: Reconfiguring daemon mon.b on smithi186 2024-03-24T11:19:15.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:15 smithi186 ceph-mon[27599]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:15.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:15 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2522431722' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:15.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:15 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:15.982 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: Reconfiguring daemon mon.c on smithi189 2024-03-24T11:19:16.812 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:16 smithi189 ceph-mon[27694]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:16.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:16.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:16.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:19:16.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:16.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:16.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:16.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: Reconfiguring daemon mon.c on smithi189 2024-03-24T11:19:16.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:16 smithi184 ceph-mon[25816]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: Reconfiguring mon.c (monmap changed)... 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:17.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: Reconfiguring daemon mon.c on smithi189 2024-03-24T11:19:17.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:16 smithi186 ceph-mon[27599]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:18.720 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-03-24T11:19:18.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:18.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:18.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:18.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:18.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:18 smithi186 ceph-mon[27599]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:18.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:18.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:18.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:18.899 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:18.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:18.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:18.900 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:18 smithi184 ceph-mon[25816]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:19.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:19.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:19.016 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:19.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.017 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:18 smithi189 ceph-mon[27694]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:19.331 DEBUG:teuthology.orchestra.run.smithi186:mgr.x> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T11:19:19.334 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T11:19:19.336 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:19:19.336 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:19:19.336 DEBUG:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:19:19.354 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:19:19.355 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/nvme1n1 2024-03-24T11:19:19.411 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/nvme1n1 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:19:01.723102807 +0000 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:16:22.014478007 +0000 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:16:22.014478007 +0000 2024-03-24T11:19:19.412 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:16:22.006478175 +0000 2024-03-24T11:19:19.412 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T11:19:19.481 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:19:19.481 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:19:19.481 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000208867 s, 2.5 MB/s 2024-03-24T11:19:19.482 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T11:19:19.539 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/nvme2n1 2024-03-24T11:19:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/nvme2n1 2024-03-24T11:19:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:19.594 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-24T11:19:19.594 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:19.594 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:19.594 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:19:01.734102576 +0000 2024-03-24T11:19:19.595 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:16:22.160474933 +0000 2024-03-24T11:19:19.595 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:16:22.160474933 +0000 2024-03-24T11:19:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:16:22.149475165 +0000 2024-03-24T11:19:19.595 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T11:19:19.660 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:19:19.661 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:19:19.661 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000183623 s, 2.8 MB/s 2024-03-24T11:19:19.662 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T11:19:19.717 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/nvme3n1 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/nvme3n1 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:19:01.746102325 +0000 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:16:22.304471901 +0000 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:16:22.304471901 +0000 2024-03-24T11:19:19.773 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:16:22.299472007 +0000 2024-03-24T11:19:19.774 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T11:19:19.839 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:19:19.839 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:19:19.839 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000207345 s, 2.5 MB/s 2024-03-24T11:19:19.840 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi184=y;smithi186=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: Saving service mgr spec with placement smithi184=y;smithi186=x;smithi189=z;count:3 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:19.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:19:19.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:19.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:19.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:19 smithi189 ceph-mon[27694]: Deploying daemon mgr.z on smithi189 2024-03-24T11:19:19.896 DEBUG:teuthology.orchestra.run.smithi184:> stat /dev/nvme4n1 2024-03-24T11:19:19.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi184=y;smithi186=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:19.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: Saving service mgr spec with placement smithi184=y;smithi186=x;smithi189=z;count:3 2024-03-24T11:19:19.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:19.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:19.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:19 smithi184 ceph-mon[25816]: Deploying daemon mgr.z on smithi189 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/nvme4n1 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2024-03-24 11:19:01.756102115 +0000 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2024-03-24 11:16:22.453468764 +0000 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2024-03-24 11:16:22.453468764 +0000 2024-03-24T11:19:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: 2024-03-24 11:16:22.442468996 +0000 2024-03-24T11:19:19.937 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T11:19:20.001 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2024-03-24T11:19:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2024-03-24T11:19:20.002 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000202856 s, 2.5 MB/s 2024-03-24T11:19:20.003 DEBUG:teuthology.orchestra.run.smithi184:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi184=y;smithi186=x;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: Saving service mgr spec with placement smithi184=y;smithi186=x;smithi189=z;count:3 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:20.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:19:20.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:20.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:20.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:19 smithi186 ceph-mon[27599]: Deploying daemon mgr.z on smithi189 2024-03-24T11:19:20.059 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:19:20.059 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:19:20.078 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:19:20.079 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/nvme1n1 2024-03-24T11:19:20.137 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/nvme1n1 2024-03-24T11:19:20.137 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:20.137 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-24T11:19:20.137 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:20.137 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:20.138 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:19:13.129457733 +0000 2024-03-24T11:19:20.138 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:16:23.634438777 +0000 2024-03-24T11:19:20.138 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:16:23.634438777 +0000 2024-03-24T11:19:20.138 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:16:23.622438988 +0000 2024-03-24T11:19:20.138 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T11:19:20.206 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:19:20.207 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:19:20.207 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000305331 s, 1.7 MB/s 2024-03-24T11:19:20.208 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T11:19:20.267 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/nvme2n1 2024-03-24T11:19:20.324 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/nvme2n1 2024-03-24T11:19:20.324 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:20.324 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-24T11:19:20.324 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:20.324 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:20.325 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:19:13.141457522 +0000 2024-03-24T11:19:20.325 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:16:23.780436218 +0000 2024-03-24T11:19:20.325 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:16:23.780436218 +0000 2024-03-24T11:19:20.325 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:16:23.769436410 +0000 2024-03-24T11:19:20.325 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T11:19:20.392 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:19:20.393 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:19:20.393 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00023495 s, 2.2 MB/s 2024-03-24T11:19:20.394 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T11:19:20.451 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/nvme3n1 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/nvme3n1 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:19:13.153457311 +0000 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:16:23.933433535 +0000 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:16:23.933433535 +0000 2024-03-24T11:19:20.509 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:16:23.925433675 +0000 2024-03-24T11:19:20.510 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T11:19:20.576 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:19:20.576 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:19:20.576 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00027748 s, 1.8 MB/s 2024-03-24T11:19:20.577 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T11:19:20.633 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/nvme4n1 2024-03-24T11:19:20.688 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/nvme4n1 2024-03-24T11:19:20.688 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:20.688 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-24T11:19:20.688 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:20.689 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:20.689 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-24 11:19:13.164457117 +0000 2024-03-24T11:19:20.689 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-24 11:16:24.082430923 +0000 2024-03-24T11:19:20.689 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-24 11:16:24.082430923 +0000 2024-03-24T11:19:20.689 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-24 11:16:24.071431115 +0000 2024-03-24T11:19:20.689 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T11:19:20.754 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-24T11:19:20.754 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-24T11:19:20.754 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000196675 s, 2.6 MB/s 2024-03-24T11:19:20.755 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T11:19:20.811 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:19:20.811 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:19:20.829 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-24T11:19:20.829 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme1n1 2024-03-24T11:19:20.885 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme1n1 2024-03-24T11:19:20.885 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:20.885 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-24T11:19:20.885 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:20.885 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:20.886 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:19:13.430845134 +0000 2024-03-24T11:19:20.886 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:16:25.277050169 +0000 2024-03-24T11:19:20.886 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:16:25.277050169 +0000 2024-03-24T11:19:20.886 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:16:25.265050394 +0000 2024-03-24T11:19:20.886 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-24T11:19:20.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:20 smithi184 ceph-mon[25816]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:20.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:20 smithi189 ceph-mon[27694]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:20.954 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:19:20.955 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:19:20.955 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000270099 s, 1.9 MB/s 2024-03-24T11:19:20.956 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-24T11:19:21.012 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme2n1 2024-03-24T11:19:21.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:20 smithi186 ceph-mon[27599]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme2n1 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:19:13.441844924 +0000 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:16:25.421047455 +0000 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:16:25.421047455 +0000 2024-03-24T11:19:21.068 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:16:25.410047663 +0000 2024-03-24T11:19:21.068 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-24T11:19:21.135 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:19:21.136 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:19:21.136 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.00086826 s, 590 kB/s 2024-03-24T11:19:21.137 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-24T11:19:21.194 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme3n1 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme3n1 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:19:13.452844715 +0000 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:16:25.573044592 +0000 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:16:25.573044592 +0000 2024-03-24T11:19:21.250 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:16:25.561044818 +0000 2024-03-24T11:19:21.251 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-24T11:19:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:19:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:19:21.320 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000204216 s, 2.5 MB/s 2024-03-24T11:19:21.321 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-24T11:19:21.377 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/nvme4n1 2024-03-24T11:19:21.432 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/nvme4n1 2024-03-24T11:19:21.432 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-24T11:19:21.432 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-03-24T11:19:21.433 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-24T11:19:21.433 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-24T11:19:21.433 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 11:19:13.463844505 +0000 2024-03-24T11:19:21.433 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-24 11:16:25.716041899 +0000 2024-03-24T11:19:21.433 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-24 11:16:25.716041899 +0000 2024-03-24T11:19:21.433 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-24 11:16:25.711041993 +0000 2024-03-24T11:19:21.433 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-24T11:19:21.500 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T11:19:21.500 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T11:19:21.500 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000197664 s, 2.6 MB/s 2024-03-24T11:19:21.502 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-24T11:19:21.559 INFO:tasks.cephadm:Deploying osd.0 on smithi184 with /dev/nvme4n1... 2024-03-24T11:19:21.560 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T11:19:21.827 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:22.116 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:21 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:21.962+0000 7fd7a54c1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: Deploying daemon mgr.x on smithi186 2024-03-24T11:19:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:22 smithi186 ceph-mon[27599]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:22.831 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:22.598+0000 7fd7a54c1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:19:22.831 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:22.689+0000 7fd7a54c1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:22.831+0000 7fd7a54c1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: Deploying daemon mgr.x on smithi186 2024-03-24T11:19:22.832 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.833 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.833 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.833 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-mon[27694]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:22.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:22.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: Deploying daemon mgr.x on smithi186 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:22.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:22 smithi184 ceph-mon[25816]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:23.115 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:22 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:22.925+0000 7fd7a54c1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:19:23.116 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:23.006+0000 7fd7a54c1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:19:23.554 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:23.266+0000 7fd7a54c1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:19:23.554 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:23.375+0000 7fd7a54c1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:23.554 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:23.459+0000 7fd7a54c1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:19:23.554 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:23.553+0000 7fd7a54c1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:19:23.720 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:23 smithi186 systemd[1]: Starting Ceph mgr.x for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:19:23.731 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T11:19:23.731 INFO:teuthology.orchestra.run.smithi184.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T11:19:23.732 INFO:teuthology.orchestra.run.smithi184.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T11:19:23.732 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10+0 records in 2024-03-24T11:19:23.732 INFO:teuthology.orchestra.run.smithi184.stderr:10+0 records out 2024-03-24T11:19:23.732 INFO:teuthology.orchestra.run.smithi184.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144194 s, 727 MB/s 2024-03-24T11:19:23.732 INFO:teuthology.orchestra.run.smithi184.stderr:--> Zapping successful for: 2024-03-24T11:19:23.865 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:23.730+0000 7fd7a54c1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:19:24.039 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:23 smithi186 podman[29288]: 2024-03-24 11:19:23.720457263 +0000 UTC m=+0.109755765 container create d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b) 2024-03-24T11:19:24.040 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:23 smithi186 podman[29288]: 2024-03-24 11:19:23.628121818 +0000 UTC m=+0.017420328 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:19:24.040 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:23 smithi186 podman[29288]: 2024-03-24 11:19:23.931262552 +0000 UTC m=+0.320561063 container init d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T11:19:24.040 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:23 smithi186 podman[29288]: 2024-03-24 11:19:23.93631566 +0000 UTC m=+0.325614179 container start d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T11:19:24.040 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:23 smithi186 ceph-mgr[29328]: -- 172.21.15.186:0/2417227545 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c7bf2a1a0 con 0x556c7bf00800 2024-03-24T11:19:24.040 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:24 smithi186 bash[29288]: d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 2024-03-24T11:19:24.292 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:24 smithi186 systemd[1]: Started Ceph mgr.x for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:19:24.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi184:/dev/nvme4n1 2024-03-24T11:19:24.594 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:19:24.748 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:24 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:24.488+0000 7f2aff976200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:19:24.748 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:24 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:24.580+0000 7f2aff976200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:19:24.954 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:24 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:24.732+0000 7fd7a54c1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:19:24.954 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:24 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:24.866+0000 7fd7a54c1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:25.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.225 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:25.225 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:19:25.225 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:25.225 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:25.225 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:25.225 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-mon[27694]: Reconfiguring daemon mgr.y on smithi184 2024-03-24T11:19:25.329 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:25.254+0000 7f2aff976200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:19:25.329 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.329 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.329 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:25.330 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-mon[27599]: Reconfiguring daemon mgr.y on smithi184 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:25.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:25 smithi184 ceph-mon[25816]: Reconfiguring daemon mgr.y on smithi184 2024-03-24T11:19:25.508 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:25.223+0000 7fd7a54c1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:19:25.508 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:25.314+0000 7fd7a54c1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:19:25.509 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:25.508+0000 7fd7a54c1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:25.596 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:25.355+0000 7f2aff976200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:19:25.597 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:25.499+0000 7f2aff976200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:19:25.597 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:25.596+0000 7f2aff976200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:19:25.794 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:25 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:25.708+0000 7fd7a54c1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:19:25.891 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:25.712+0000 7f2aff976200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:19:26.102 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:26 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:26.003+0000 7fd7a54c1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:26.201 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:25 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:25.987+0000 7f2aff976200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:19:26.201 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:26 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:26.102+0000 7f2aff976200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:26.201 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:26 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:26.200+0000 7f2aff976200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:19:26.365 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:26 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:26.289+0000 7fd7a54c1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:26.496 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:26 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:26.497 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:26 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:26.310+0000 7f2aff976200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:19:26.497 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:26 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:26.495+0000 7f2aff976200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:19:26.637 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:26 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:26.637 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:26 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:26.543+0000 7fd7a54c1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:19:26.659 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:26 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.615 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:27.379+0000 7fd7a54c1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.660 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:27.541+0000 7f2aff976200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:19:27.660 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:27.660 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.660 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.660 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:27.660 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:27.661 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:27.661 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.661 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.661 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.661 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.661 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:27.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.014 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:27 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:27.678+0000 7f2aff976200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:19:28.015 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:28.014+0000 7f2aff976200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:19:28.069 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:27 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:27.971+0000 7fd7a54c1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:19:28.069 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:28.069+0000 7fd7a54c1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:19:28.292 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:28.102+0000 7f2aff976200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:19:28.365 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:19:28.283+0000 7fd7a54c1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: Standby manager daemon z started 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:19:28.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:28 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:28.749 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:28.292+0000 7f2aff976200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:28.749 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:28.472+0000 7f2aff976200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: Standby manager daemon z started 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:19:28.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: Standby manager daemon z started 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:19:28.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:28 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/2986411982' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:29.008 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:28 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:28.748+0000 7f2aff976200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:19:29.008 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:29 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:29.008+0000 7f2aff976200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:19:29.292 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:29 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:29.238+0000 7f2aff976200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:19:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:29 smithi184 ceph-mon[25816]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:29 smithi184 ceph-mon[25816]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:29 smithi184 ceph-mon[25816]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T11:19:29.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:29 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:19:29.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:29 smithi186 ceph-mon[27599]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:29.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:29 smithi186 ceph-mon[27599]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:29.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:29 smithi186 ceph-mon[27599]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T11:19:29.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:29 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:19:29.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:29 smithi189 ceph-mon[27694]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi184:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:29.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:29 smithi189 ceph-mon[27694]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:29.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:29 smithi189 ceph-mon[27694]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-24T11:19:29.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:29 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:19:30.291 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:30.022+0000 7f2aff976200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:19:30.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:30 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/4091644830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb30c42f-c967-489d-95fd-3abe16af1b45"}]: dispatch 2024-03-24T11:19:30.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:30 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/4091644830' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb30c42f-c967-489d-95fd-3abe16af1b45"}]': finished 2024-03-24T11:19:30.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:30 smithi184 ceph-mon[25816]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:19:30.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:30 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:30.683 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/4091644830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb30c42f-c967-489d-95fd-3abe16af1b45"}]: dispatch 2024-03-24T11:19:30.683 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/4091644830' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb30c42f-c967-489d-95fd-3abe16af1b45"}]': finished 2024-03-24T11:19:30.683 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-mon[27599]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:19:30.683 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:30.684 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:30.587+0000 7f2aff976200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:19:30.684 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:30.682+0000 7f2aff976200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:19:30.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:30 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/4091644830' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb30c42f-c967-489d-95fd-3abe16af1b45"}]: dispatch 2024-03-24T11:19:30.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:30 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/4091644830' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb30c42f-c967-489d-95fd-3abe16af1b45"}]': finished 2024-03-24T11:19:30.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:30 smithi189 ceph-mon[27694]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T11:19:30.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:30 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:31.042 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:19:30 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:19:30.893+0000 7f2aff976200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: Standby manager daemon x started 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:31.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:31 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1849859632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: Standby manager daemon x started 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:31 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1849859632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: Standby manager daemon x started 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/3011590187' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:19:31.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:31 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1849859632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:19:32.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:32 smithi186 ceph-mon[27599]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T11:19:32.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:32 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:19:32.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:32 smithi184 ceph-mon[25816]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T11:19:32.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:32 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:19:32.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:32 smithi189 ceph-mon[27694]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-24T11:19:32.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:32 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:19:33.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:33 smithi184 ceph-mon[25816]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:33.685 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:33 smithi189 ceph-mon[27694]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:33.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:33 smithi186 ceph-mon[27599]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:35 smithi184 ceph-mon[25816]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:35.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:35 smithi186 ceph-mon[27599]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:35.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:35 smithi189 ceph-mon[27694]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:37 smithi184 ceph-mon[25816]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:19:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:37.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:37 smithi186 ceph-mon[27599]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:37.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:19:37.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:37.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:37 smithi189 ceph-mon[27694]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:37.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:19:37.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:38.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:38 smithi184 ceph-mon[25816]: Deploying daemon osd.0 on smithi184 2024-03-24T11:19:38.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:38 smithi186 ceph-mon[27599]: Deploying daemon osd.0 on smithi184 2024-03-24T11:19:38.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:38 smithi189 ceph-mon[27694]: Deploying daemon osd.0 on smithi184 2024-03-24T11:19:39.593 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:39 smithi184 ceph-mon[25816]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:39.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:39 smithi186 ceph-mon[27599]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:39.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:39 smithi189 ceph-mon[27694]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:41.626 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:41 smithi184 ceph-mon[25816]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:41.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:41 smithi186 ceph-mon[27599]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:41.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:41 smithi189 ceph-mon[27694]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:43.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:43.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:43.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:19:43.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:19:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:19:43.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:44.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: Reconfiguring osd.0 (unknown last config time)... 2024-03-24T11:19:44.206 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: Reconfiguring daemon osd.0 on smithi184 2024-03-24T11:19:44.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:19:44.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:44.207 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: Reconfiguring osd.0 (unknown last config time)... 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: Reconfiguring daemon osd.0 on smithi184 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:19:44.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:44.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: Reconfiguring osd.0 (unknown last config time)... 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: Reconfiguring daemon osd.0 on smithi184 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:44.355 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.167 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:45 smithi184 ceph-mon[25816]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:46.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.168 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.178 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:45 smithi189 ceph-mon[27694]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:46.178 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:45 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.178 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:45 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.178 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:45 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.244 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:45 smithi186 ceph-mon[27599]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:46.244 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:45 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.244 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:45 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:46.244 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:45 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:47.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: Cluster is now healthy 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.776 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: Cluster is now healthy 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:47.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: Cluster is now healthy 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:47.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:47.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:47.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:47.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:48.695 INFO:teuthology.orchestra.run.smithi184.stdout:Created osd(s) 0 on host 'smithi184' 2024-03-24T11:19:49.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:49.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:19:49.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:49.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.218 DEBUG:teuthology.orchestra.run.smithi184:osd.0> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:19:49.220 INFO:tasks.cephadm:Deploying osd.1 on smithi186 with /dev/nvme4n1... 2024-03-24T11:19:49.220 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:49.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:49.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:19:49.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:19:49.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]: dispatch 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:19:49.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:19:49.469 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:49.671 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0[38383]: 2024-03-24T11:19:49.352+0000 7fab57796700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T11:19:49.671 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0[38383]: 2024-03-24T11:19:49.357+0000 7fab4f581700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: purged_snaps scrub starts 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: purged_snaps scrub ok 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: Adjusting osd_memory_target on smithi184 to 16064M 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' 2024-03-24T11:19:50.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:49 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: purged_snaps scrub starts 2024-03-24T11:19:50.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: purged_snaps scrub ok 2024-03-24T11:19:50.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: Adjusting osd_memory_target on smithi184 to 16064M 2024-03-24T11:19:50.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-03-24T11:19:50.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:19:50.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' 2024-03-24T11:19:50.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:49 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: purged_snaps scrub starts 2024-03-24T11:19:50.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: purged_snaps scrub ok 2024-03-24T11:19:50.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: Adjusting osd_memory_target on smithi184 to 16064M 2024-03-24T11:19:50.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi184", "root=default"]}]': finished 2024-03-24T11:19:50.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:19:50.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:50.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: from='osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605]' entity='osd.0' 2024-03-24T11:19:50.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:49 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125385 s, 836 MB/s 2024-03-24T11:19:51.393 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-24T11:19:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:51 smithi184 ceph-mon[25816]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:51 smithi184 ceph-mon[25816]: osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605] boot 2024-03-24T11:19:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:51 smithi184 ceph-mon[25816]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:19:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:51 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:51.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:51 smithi186 ceph-mon[27599]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:51.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:51 smithi186 ceph-mon[27599]: osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605] boot 2024-03-24T11:19:51.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:51 smithi186 ceph-mon[27599]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:19:51.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:51 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:51.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:51 smithi189 ceph-mon[27694]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:19:51.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:51 smithi189 ceph-mon[27694]: osd.0 [v2:172.21.15.184:6802/3760044605,v1:172.21.15.184:6803/3760044605] boot 2024-03-24T11:19:51.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:51 smithi189 ceph-mon[27694]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T11:19:51.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:51 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:19:51.973 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi186:/dev/nvme4n1 2024-03-24T11:19:52.261 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:19:52.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:52 smithi186 ceph-mon[27599]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:19:52.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:52 smithi184 ceph-mon[25816]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:19:52.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:52 smithi189 ceph-mon[27694]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:19:53.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:53 smithi186 ceph-mon[27599]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:53.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:53 smithi184 ceph-mon[25816]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:53.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:53 smithi189 ceph-mon[27694]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:54.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:54 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:19:54.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:54 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:19:54.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:54 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:54.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:54 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:19:54.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:54 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:19:54.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:54 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:54.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:54 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:19:54.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:54 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:19:54.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:54 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:19:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:55 smithi186 ceph-mon[27599]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:55 smithi186 ceph-mon[27599]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:55.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:55 smithi184 ceph-mon[25816]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:55.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:55 smithi184 ceph-mon[25816]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:55.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:55 smithi189 ceph-mon[27694]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:55.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:55 smithi189 ceph-mon[27694]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:19:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:57 smithi184 ceph-mon[25816]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:57 smithi184 ceph-mon[25816]: from='client.? 172.21.15.186:0/2384178576' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]: dispatch 2024-03-24T11:19:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:57 smithi184 ceph-mon[25816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]: dispatch 2024-03-24T11:19:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:57 smithi184 ceph-mon[25816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]': finished 2024-03-24T11:19:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:57 smithi184 ceph-mon[25816]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:19:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:57 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:19:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:57 smithi186 ceph-mon[27599]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:57 smithi186 ceph-mon[27599]: from='client.? 172.21.15.186:0/2384178576' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]: dispatch 2024-03-24T11:19:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:57 smithi186 ceph-mon[27599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]: dispatch 2024-03-24T11:19:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:57 smithi186 ceph-mon[27599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]': finished 2024-03-24T11:19:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:57 smithi186 ceph-mon[27599]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:19:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:57 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:19:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:57 smithi189 ceph-mon[27694]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:57 smithi189 ceph-mon[27694]: from='client.? 172.21.15.186:0/2384178576' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]: dispatch 2024-03-24T11:19:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:57 smithi189 ceph-mon[27694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]: dispatch 2024-03-24T11:19:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:57 smithi189 ceph-mon[27694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c102f5-1b96-4121-b771-85a4eb8fc5eb"}]': finished 2024-03-24T11:19:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:57 smithi189 ceph-mon[27694]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T11:19:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:57 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:19:58.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:58 smithi184 ceph-mon[25816]: from='client.? 172.21.15.186:0/3335300500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:19:58.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:58 smithi186 ceph-mon[27599]: from='client.? 172.21.15.186:0/3335300500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:19:58.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:58 smithi189 ceph-mon[27694]: from='client.? 172.21.15.186:0/3335300500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:19:59.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:19:59 smithi184 ceph-mon[25816]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:59.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:19:59 smithi186 ceph-mon[27599]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:19:59.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:19:59 smithi189 ceph-mon[27694]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:00.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:00 smithi184 ceph-mon[25816]: overall HEALTH_OK 2024-03-24T11:20:00.791 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:00 smithi186 ceph-mon[27599]: overall HEALTH_OK 2024-03-24T11:20:00.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:00 smithi189 ceph-mon[27694]: overall HEALTH_OK 2024-03-24T11:20:01.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:01 smithi186 ceph-mon[27599]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:01.654 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:01 smithi184 ceph-mon[25816]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:01.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:01 smithi189 ceph-mon[27694]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:03.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:03.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:20:03.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:03 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:20:03.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:03 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:20:03.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:03 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: Detected new or changed devices on smithi184 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: Deploying daemon osd.1 on smithi186 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.525 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: Detected new or changed devices on smithi184 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: Deploying daemon osd.1 on smithi186 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:04.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:04.527 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:04.527 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: Detected new or changed devices on smithi184 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: Deploying daemon osd.1 on smithi186 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:04.794 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:04.795 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:04.795 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:04.795 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:05 smithi189 ceph-mon[27694]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:05 smithi189 ceph-mon[27694]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:20:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:05 smithi189 ceph-mon[27694]: Cluster is now healthy 2024-03-24T11:20:05.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:05 smithi184 ceph-mon[25816]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:05.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:05 smithi184 ceph-mon[25816]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:20:05.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:05 smithi184 ceph-mon[25816]: Cluster is now healthy 2024-03-24T11:20:05.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:05 smithi186 ceph-mon[27599]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:05.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:05 smithi186 ceph-mon[27599]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-24T11:20:05.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:05 smithi186 ceph-mon[27599]: Cluster is now healthy 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.229 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:07.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:09.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:09 smithi186 ceph-mon[27599]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:09.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:09 smithi189 ceph-mon[27694]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:09.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:09 smithi184 ceph-mon[25816]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:10 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:10 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:11.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:11.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:11.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:11.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:11.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:10 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:13.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:12 smithi184 ceph-mon[25816]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:13.288 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:12 smithi186 ceph-mon[27599]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:13.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:12 smithi189 ceph-mon[27694]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:13.611 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 1 on host 'smithi186' 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:14.159 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:13 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:20:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:20:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:14.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:14.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:14.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:14.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:13 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.184 DEBUG:teuthology.orchestra.run.smithi186:osd.1> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:20:14.186 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/nvme4n1... 2024-03-24T11:20:14.186 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-24T11:20:14.217 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:20:14.217 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:20:14.217 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.217 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.217 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:14.218 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:14.218 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.218 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:14.218 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:14.218 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:13 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:14.435 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:20:15.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: Adjusting osd_memory_target on smithi186 to 16064M 2024-03-24T11:20:15.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:15.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:20:15.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: from='osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-24T11:20:15.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:20:15.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:15.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:14 smithi189 ceph-mon[27694]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: Adjusting osd_memory_target on smithi186 to 16064M 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: from='osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:14 smithi186 ceph-mon[27599]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-24T11:20:15.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: Adjusting osd_memory_target on smithi186 to 16064M 2024-03-24T11:20:15.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:15.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:20:15.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: from='osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-24T11:20:15.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:20:15.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:15.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:14 smithi184 ceph-mon[25816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-24T11:20:15.792 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1[33197]: 2024-03-24T11:20:15.396+0000 7f7ef7fad700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:20:15.792 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1[33197]: 2024-03-24T11:20:15.399+0000 7f7eefd98700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: purged_snaps scrub starts 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: purged_snaps scrub ok 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:15 smithi189 ceph-mon[27694]: from='osd.1 ' entity='osd.1' 2024-03-24T11:20:16.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: purged_snaps scrub starts 2024-03-24T11:20:16.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: purged_snaps scrub ok 2024-03-24T11:20:16.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-24T11:20:16.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:20:16.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:15 smithi186 ceph-mon[27599]: from='osd.1 ' entity='osd.1' 2024-03-24T11:20:16.370 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/nvme4n1 2024-03-24T11:20:16.370 INFO:teuthology.orchestra.run.smithi189.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-24T11:20:16.370 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-24T11:20:16.371 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-24T11:20:16.371 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-24T11:20:16.371 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130399 s, 804 MB/s 2024-03-24T11:20:16.371 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: purged_snaps scrub starts 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: purged_snaps scrub ok 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:15 smithi184 ceph-mon[25816]: from='osd.1 ' entity='osd.1' 2024-03-24T11:20:16.973 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:16 smithi189 ceph-mon[27694]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:16.973 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:16.973 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:16 smithi189 ceph-mon[27694]: osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296] boot 2024-03-24T11:20:16.973 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:16 smithi189 ceph-mon[27694]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:20:16.973 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:17.001 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi189:/dev/nvme4n1 2024-03-24T11:20:17.279 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:20:17.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:16 smithi186 ceph-mon[27599]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:17.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:17.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:16 smithi186 ceph-mon[27599]: osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296] boot 2024-03-24T11:20:17.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:16 smithi186 ceph-mon[27599]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:20:17.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:16 smithi184 ceph-mon[25816]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:20:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:16 smithi184 ceph-mon[25816]: osd.1 [v2:172.21.15.186:6800/682526296,v1:172.21.15.186:6801/682526296] boot 2024-03-24T11:20:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:16 smithi184 ceph-mon[25816]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T11:20:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:20:19.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:18 smithi186 ceph-mon[27599]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:19.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:18 smithi186 ceph-mon[27599]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:20:19.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:18 smithi189 ceph-mon[27694]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:19.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:18 smithi189 ceph-mon[27694]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:20:19.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:18 smithi184 ceph-mon[25816]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:19.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:18 smithi184 ceph-mon[25816]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:20:19.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:19 smithi189 ceph-mon[27694]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:19.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:20:19.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:20:19.967 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:19 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:20.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:19 smithi186 ceph-mon[27599]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:20.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:20:20.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:20:20.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:19 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:20.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:19 smithi184 ceph-mon[25816]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:20.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:20:20.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:20:20.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:19 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:21.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:20 smithi189 ceph-mon[27694]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:21.193 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:20 smithi186 ceph-mon[27599]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:21.216 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:20 smithi184 ceph-mon[25816]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:22.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:21 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/1167411163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]: dispatch 2024-03-24T11:20:22.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:21 smithi184 ceph-mon[25816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]: dispatch 2024-03-24T11:20:22.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:21 smithi184 ceph-mon[25816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]': finished 2024-03-24T11:20:22.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:21 smithi184 ceph-mon[25816]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:20:22.235 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:21 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:21 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/1167411163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]: dispatch 2024-03-24T11:20:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:21 smithi189 ceph-mon[27694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]: dispatch 2024-03-24T11:20:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:21 smithi189 ceph-mon[27694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]': finished 2024-03-24T11:20:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:21 smithi189 ceph-mon[27694]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:20:22.236 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:21 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:22.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:21 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/1167411163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]: dispatch 2024-03-24T11:20:22.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:21 smithi186 ceph-mon[27599]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]: dispatch 2024-03-24T11:20:22.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:21 smithi186 ceph-mon[27599]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17cd9e6d-11f8-4c70-8cab-b26eec2df1c5"}]': finished 2024-03-24T11:20:22.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:21 smithi186 ceph-mon[27599]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T11:20:22.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:21 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:23.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:22 smithi184 ceph-mon[25816]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:23.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.260 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:22 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:22 smithi186 ceph-mon[27599]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:23.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:22 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:22 smithi189 ceph-mon[27694]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:23.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:23.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:22 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:24.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:23 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/3068027451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:20:24.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:23 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/3068027451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:20:24.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:23 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/3068027451' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:20:25.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:24 smithi186 ceph-mon[27599]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:25.354 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:24 smithi189 ceph-mon[27694]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:25.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:24 smithi184 ceph-mon[25816]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:27 smithi184 ceph-mon[25816]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:27 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:27 smithi186 ceph-mon[27599]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:27.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:27 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:27 smithi189 ceph-mon[27694]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:27 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:29 smithi189 ceph-mon[27694]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:29.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:29 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:29 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:29 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:29 smithi184 ceph-mon[25816]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:29.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:29 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:29 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:29 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.438 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:29 smithi186 ceph-mon[27599]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:29.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:29 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:29 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:29.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:29 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:30.378 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:30 smithi186 ceph-mon[27599]: Detected new or changed devices on smithi186 2024-03-24T11:20:30.437 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:30 smithi189 ceph-mon[27694]: Detected new or changed devices on smithi186 2024-03-24T11:20:30.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:30 smithi184 ceph-mon[25816]: Detected new or changed devices on smithi186 2024-03-24T11:20:31.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:31 smithi189 ceph-mon[27694]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:31.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:31 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:20:31.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:31 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:31.321 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:31 smithi189 ceph-mon[27694]: Deploying daemon osd.2 on smithi189 2024-03-24T11:20:31.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:31 smithi186 ceph-mon[27599]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:31.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:31 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:20:31.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:31 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:31.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:31 smithi186 ceph-mon[27599]: Deploying daemon osd.2 on smithi189 2024-03-24T11:20:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:31 smithi184 ceph-mon[25816]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:31 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:20:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:31 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:31.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:31 smithi184 ceph-mon[25816]: Deploying daemon osd.2 on smithi189 2024-03-24T11:20:33.291 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:33 smithi189 ceph-mon[27694]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:33.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:33 smithi186 ceph-mon[27599]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:33.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:33 smithi184 ceph-mon[25816]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:35.448 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:35 smithi189 ceph-mon[27694]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:35.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:35 smithi186 ceph-mon[27599]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:35.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:35 smithi184 ceph-mon[25816]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:37.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:37 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:37.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:37.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:37.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:37.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:37 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:37.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:37 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:39.070 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-03-24T11:20:39.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:39 smithi186 ceph-mon[27599]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:39.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:39 smithi189 ceph-mon[27694]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:39.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:39 smithi184 ceph-mon[25816]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:39.684 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:20:39.687 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-24T11:20:39.687 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:20:40.169 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: Adjusting osd_memory_target on smithi189 to 16064M 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:40.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:20:40.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:40 smithi184 ceph-mon[25816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: Adjusting osd_memory_target on smithi189 to 16064M 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:40.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:40.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:20:40.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:40 smithi186 ceph-mon[27599]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: Adjusting osd_memory_target on smithi189 to 16064M 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:40.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:40.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:40.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:20:40.367 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:40 smithi189 ceph-mon[27694]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:20:41.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:41 smithi184 ceph-mon[25816]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:41.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:41 smithi184 ceph-mon[25816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:20:41.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:41 smithi184 ceph-mon[25816]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:20:41.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:41 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:41.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:41 smithi184 ceph-mon[25816]: from='osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:20:41.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:41 smithi184 ceph-mon[25816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:20:41.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-mon[27694]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:41.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-mon[27694]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:20:41.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-mon[27694]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:20:41.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:41.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-mon[27694]: from='osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:20:41.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-mon[27694]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:20:41.366 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2[34587]: 2024-03-24T11:20:41.070+0000 7fc556799700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:20:41.367 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:41 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2[34587]: 2024-03-24T11:20:41.073+0000 7fc55258c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:20:41.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:41 smithi186 ceph-mon[27599]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:41.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:41 smithi186 ceph-mon[27599]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:20:41.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:41 smithi186 ceph-mon[27599]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:20:41.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:41 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:41.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:41 smithi186 ceph-mon[27599]: from='osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:20:41.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:41 smithi186 ceph-mon[27599]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T11:20:42.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:42 smithi189 ceph-mon[27694]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T11:20:42.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:42 smithi189 ceph-mon[27694]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:20:42.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:42.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:42 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:42.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:42 smithi184 ceph-mon[25816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T11:20:42.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:42 smithi184 ceph-mon[25816]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:20:42.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:42.401 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:42 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:42.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:42 smithi186 ceph-mon[27599]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T11:20:42.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:42 smithi186 ceph-mon[27599]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:20:42.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:42.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:42 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: purged_snaps scrub starts 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: purged_snaps scrub ok 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320] boot 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:43 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: purged_snaps scrub starts 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: purged_snaps scrub ok 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320] boot 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:43.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:20:43.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:43 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.447 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: purged_snaps scrub starts 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: purged_snaps scrub ok 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: osd.2 [v2:172.21.15.189:6800/3420409320,v1:172.21.15.189:6801/3420409320] boot 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:43.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:43 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:44.011 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":19,"num_osds":3,"num_up_osds":3,"osd_up_since":1711279242,"num_in_osds":3,"osd_in_since":1711279221,"num_remapped_pgs":0} 2024-03-24T11:20:44.011 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:20:44.259 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:44.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:44 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:20:44.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:44 smithi189 ceph-mon[27694]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:20:44.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:44 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:20:44.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:44 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/606266674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:20:44.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:44 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:20:44.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:44 smithi184 ceph-mon[25816]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:20:44.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:44 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:20:44.380 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:44 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/606266674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:20:44.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:44 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T11:20:44.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:44 smithi186 ceph-mon[27599]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:20:44.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:44 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:20:44.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:44 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/606266674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:20:44.671 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40908]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T11:20:44.672 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40908]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:44.672 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40908]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:44.672 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40908]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:44.966 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40949]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T11:20:44.966 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40949]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:44.966 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40949]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:44.966 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:20:44 smithi184 sudo[40949]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:45.219 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:44 smithi186 sudo[34720]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T11:20:45.219 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:44 smithi186 sudo[34720]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:45.219 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:44 smithi186 sudo[34720]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:45.219 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:44 smithi186 sudo[34720]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:45.219 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:45 smithi186 ceph-mon[27599]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:45.219 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:45 smithi186 ceph-mon[27599]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:45.219 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:45 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:20:45.220 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:45 smithi186 ceph-mon[27599]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:20:45.220 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:45 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:45.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:45 smithi184 ceph-mon[25816]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:45.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:45 smithi184 ceph-mon[25816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:45.283 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:20:45.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:45 smithi184 ceph-mon[25816]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:20:45.284 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:45 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:45.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:45 smithi189 ceph-mon[27694]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:45.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:45 smithi189 ceph-mon[27694]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:45.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:45 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:20:45.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:45 smithi189 ceph-mon[27694]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:20:45.366 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:45 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:45.542 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:45 smithi186 sudo[34723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T11:20:45.542 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:45 smithi186 sudo[34723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:45.542 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:45 smithi186 sudo[34723]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:45.542 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:20:45 smithi186 sudo[34723]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:45.740 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35641]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-24T11:20:45.740 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35641]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:45.740 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35641]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:45.741 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35641]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:46.115 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-24T11:20:46.116 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:46.116 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35644]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:46.116 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:20:45 smithi189 sudo[35644]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:46.258 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:20:46.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:46.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:46.259 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41017]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:46.478 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:20:46.479 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":21,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:17:13.767382+0000","modified":"2024-03-24T11:20:45.087339+0000","last_up_change":"2024-03-24T11:20:42.067203+0000","last_in_change":"2024-03-24T11:20:21.821174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:20:42.614362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb30c42f-c967-489d-95fd-3abe16af1b45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6803","nonce":3760044605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6805","nonce":3760044605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6809","nonce":3760044605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6807","nonce":3760044605}]},"public_addr":"172.21.15.184:6803/3760044605","cluster_addr":"172.21.15.184:6805/3760044605","heartbeat_back_addr":"172.21.15.184:6809/3760044605","heartbeat_front_addr":"172.21.15.184:6807/3760044605","state":["exists","up"]},{"osd":1,"uuid":"83c102f5-1b96-4121-b771-85a4eb8fc5eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6801","nonce":682526296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6803","nonce":682526296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6807","nonce":682526296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6805","nonce":682526296}]},"public_addr":"172.21.15.186:6801/682526296","cluster_addr":"172.21.15.186:6803/682526296","heartbeat_back_addr":"172.21.15.186:6807/682526296","heartbeat_front_addr":"172.21.15.186:6805/682526296","state":["exists","up"]},{"osd":2,"uuid":"17cd9e6d-11f8-4c70-8cab-b26eec2df1c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3420409320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3420409320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3420409320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3420409320}]},"public_addr":"172.21.15.189:6801/3420409320","cluster_addr":"172.21.15.189:6803/3420409320","heartbeat_back_addr":"172.21.15.189:6807/3420409320","heartbeat_front_addr":"172.21.15.189:6805/3420409320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:47.901976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:20:13.957580+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:20:40.084873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:0/3013664472":"2024-03-25T11:18:10.276568+0000","172.21.15.184:6800/4057035916":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/1363820209":"2024-03-25T11:17:43.481718+0000","172.21.15.184:6800/873414609":"2024-03-25T11:17:43.481718+0000","172.21.15.184:6801/4057035916":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/2401055791":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/3563561133":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/2143438781":"2024-03-25T11:18:10.276568+0000","172.21.15.184:6801/873414609":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/544235517":"2024-03-25T11:17:43.481718+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:20:46.532 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 ceph-mon[27694]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:20:46.532 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:46.532 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:46.532 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 ceph-mon[27694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:46.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:46 smithi186 ceph-mon[27599]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:20:46.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:46 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:46.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:46 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:46.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:46 smithi186 ceph-mon[27599]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41020]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 sudo[41020]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 ceph-mon[25816]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:46.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:46.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:46 smithi184 ceph-mon[25816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:46.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35708]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:20:46.786 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35708]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:46.786 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35708]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:46.786 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35708]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:47.049 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:20:42.614362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T11:20:47.049 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:20:47.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35711]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:20:47.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35711]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:47.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35711]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:47.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:46 smithi189 sudo[35711]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:47.286 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:20:47.301 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:20:47.301 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:47.301 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34726]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:47.301 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34726]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T11:20:47.408 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: Detected new or changed devices on smithi189 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1881691021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: Cluster is now healthy 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:47.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:47 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T11:20:47.547 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: Detected new or changed devices on smithi189 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1881691021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: Cluster is now healthy 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:47.548 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.549 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.549 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.549 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:47 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34729]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34729]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34729]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 sudo[34729]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:47.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: Detected new or changed devices on smithi189 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1881691021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:20:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: Cluster is now healthy 2024-03-24T11:20:47.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:47.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:47.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:20:47.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:20:47.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:47 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:20:48.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:48 smithi184 ceph-mon[25816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:48.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:48 smithi184 ceph-mon[25816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:48.356 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.357 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:48 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.435 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:48 smithi189 ceph-mon[27694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:48.436 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:48 smithi189 ceph-mon[27694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:48.436 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.436 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.436 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.436 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:48 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.584 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:48 smithi186 ceph-mon[27599]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:20:48.584 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:48 smithi186 ceph-mon[27599]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:20:48.584 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.584 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.584 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:48.584 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:48 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:49.088 INFO:teuthology.orchestra.run.smithi184.stdout:pg_num: 1 2024-03-24T11:20:49.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:49 smithi184 ceph-mon[25816]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:49.395 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:49 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3517287693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:20:49.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:49 smithi189 ceph-mon[27694]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:49.490 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:49 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3517287693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:20:49.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:49 smithi186 ceph-mon[27599]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:49.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:49 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3517287693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:20:49.681 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi184 2024-03-24T11:20:49.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi184=foo.a' 2024-03-24T11:20:49.926 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:20:51.475 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:51 smithi189 ceph-mon[27694]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:51.657 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled rgw.foo.a update... 2024-03-24T11:20:51.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:51 smithi184 ceph-mon[25816]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:51.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:51 smithi186 ceph-mon[27599]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:52.227 DEBUG:teuthology.orchestra.run.smithi184:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T11:20:52.230 INFO:tasks.cephadm:Adding prometheus.a on smithi186 2024-03-24T11:20:52.230 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi186=a' 2024-03-24T11:20:52.469 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi184=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: Saving service rgw.foo.a spec with placement smithi184=foo.a;count:1 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: Deploying daemon rgw.foo.a on smithi184 2024-03-24T11:20:52.867 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:52 smithi189 ceph-mon[27694]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:52.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi184=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:52.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: Saving service rgw.foo.a spec with placement smithi184=foo.a;count:1 2024-03-24T11:20:52.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:52.910 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: Deploying daemon rgw.foo.a on smithi184 2024-03-24T11:20:52.911 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:52 smithi184 ceph-mon[25816]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi184=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: Saving service rgw.foo.a spec with placement smithi184=foo.a;count:1 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T11:20:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T11:20:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: Deploying daemon rgw.foo.a on smithi184 2024-03-24T11:20:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:52 smithi186 ceph-mon[27599]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:20:53.798 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:53 smithi184 systemd[1]: Starting Ceph rgw.foo.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:20:54.115 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:53 smithi184 podman[41571]: 2024-03-24 11:20:53.700459867 +0000 UTC m=+0.018526873 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:20:54.115 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:53 smithi184 podman[41571]: 2024-03-24 11:20:53.804190281 +0000 UTC m=+0.122257271 container create e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T11:20:54.115 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:54 smithi184 podman[41571]: 2024-03-24 11:20:54.006876069 +0000 UTC m=+0.324943071 container init e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T11:20:54.116 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:54 smithi184 podman[41571]: 2024-03-24 11:20:54.012362577 +0000 UTC m=+0.330429568 container start e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T11:20:54.116 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:54 smithi184 bash[41571]: e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab 2024-03-24T11:20:54.296 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled prometheus update... 2024-03-24T11:20:54.421 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:20:54 smithi184 systemd[1]: Started Ceph rgw.foo.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:20:54.872 DEBUG:teuthology.orchestra.run.smithi186:prometheus.a> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T11:20:54.875 INFO:tasks.cephadm:Adding node-exporter.a on smithi184 2024-03-24T11:20:54.875 INFO:tasks.cephadm:Adding node-exporter.b on smithi186 2024-03-24T11:20:54.875 INFO:tasks.cephadm:Adding node-exporter.c on smithi189 2024-03-24T11:20:54.875 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '3;smithi184=a;smithi186=b;smithi189=c' 2024-03-24T11:20:55.110 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:20:55.409 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: Saving service rgw.foo.a spec with placement smithi184=foo.a;count:1 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: Checking dashboard <-> RGW credentials 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:20:55.410 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3851784611' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:20:55.411 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:55 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/553798409' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: Saving service rgw.foo.a spec with placement smithi184=foo.a;count:1 2024-03-24T11:20:55.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: Checking dashboard <-> RGW credentials 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3851784611' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:20:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:55 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/553798409' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: Saving service rgw.foo.a spec with placement smithi184=foo.a;count:1 2024-03-24T11:20:55.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: Checking dashboard <-> RGW credentials 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3851784611' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:20:55.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:55 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/553798409' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T11:20:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:56 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:56 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3851784611' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:20:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:56 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/553798409' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:20:56.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:56 smithi184 ceph-mon[25816]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:20:56.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:56 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:56.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:56 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3851784611' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:20:56.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:56 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/553798409' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:20:56.730 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:56 smithi189 ceph-mon[27694]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:20:56.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:56 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:56.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:56 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3851784611' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:20:56.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:56 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/553798409' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T11:20:56.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:56 smithi186 ceph-mon[27599]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:20:56.955 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled node-exporter update... 2024-03-24T11:20:57.579 DEBUG:teuthology.orchestra.run.smithi184:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T11:20:57.582 DEBUG:teuthology.orchestra.run.smithi186:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T11:20:57.585 DEBUG:teuthology.orchestra.run.smithi189:node-exporter.c> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.c.service 2024-03-24T11:20:57.587 INFO:tasks.cephadm:Adding alertmanager.a on smithi184 2024-03-24T11:20:57.587 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi184=a' 2024-03-24T11:20:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: pgmap v102: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:20:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T11:20:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi184=a;smithi186=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:57.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: Saving service node-exporter spec with placement smithi184=a;smithi186=b;smithi189=c;count:3 2024-03-24T11:20:57.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:57 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: pgmap v102: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi184=a;smithi186=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: Saving service node-exporter spec with placement smithi184=a;smithi186=b;smithi189=c;count:3 2024-03-24T11:20:57.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:57 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:57.859 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:20:57.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: pgmap v102: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-24T11:20:57.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: osdmap e24: 3 total, 3 up, 3 in 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi184=a;smithi186=b;smithi189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: Saving service node-exporter spec with placement smithi184=a;smithi186=b;smithi189=c;count:3 2024-03-24T11:20:57.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:57 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: pgmap v105: 65 pgs: 25 active+clean, 20 creating+peering, 8 creating+activating, 12 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.3 KiB/s rd, 131 KiB/s wr, 12 op/s 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:58 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: pgmap v105: 65 pgs: 25 active+clean, 20 creating+peering, 8 creating+activating, 12 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.3 KiB/s rd, 131 KiB/s wr, 12 op/s 2024-03-24T11:20:59.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:20:59.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:20:59.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T11:20:59.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:58 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: pgmap v105: 65 pgs: 25 active+clean, 20 creating+peering, 8 creating+activating, 12 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.3 KiB/s rd, 131 KiB/s wr, 12 op/s 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: osdmap e25: 3 total, 3 up, 3 in 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:58 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:20:59.605 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled alertmanager update... 2024-03-24T11:21:00.187 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:59 smithi189 ceph-mon[27694]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T11:21:00.187 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:59 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.187 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:59 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.187 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:59 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.187 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:20:59 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:00.213 DEBUG:teuthology.orchestra.run.smithi184:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T11:21:00.215 INFO:tasks.cephadm:Adding grafana.a on smithi186 2024-03-24T11:21:00.216 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi186=a' 2024-03-24T11:21:00.239 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:59 smithi184 ceph-mon[25816]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T11:21:00.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:59 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:59 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:59 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.240 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:20:59 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:00.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:59 smithi186 ceph-mon[27599]: osdmap e26: 3 total, 3 up, 3 in 2024-03-24T11:21:00.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:59 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:59 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:59 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T11:21:00.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:20:59 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:21:01.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:00 smithi189 ceph-mon[27694]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:21:01.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:00 smithi189 ceph-mon[27694]: Saving service alertmanager spec with placement smithi184=a;count:1 2024-03-24T11:21:01.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:00 smithi189 ceph-mon[27694]: pgmap v108: 97 pgs: 25 active+clean, 20 creating+peering, 8 creating+activating, 44 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-24T11:21:01.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:00 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:21:01.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:00 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:21:01.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:00 smithi189 ceph-mon[27694]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T11:21:01.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:00 smithi186 ceph-mon[27599]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:21:01.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:00 smithi186 ceph-mon[27599]: Saving service alertmanager spec with placement smithi184=a;count:1 2024-03-24T11:21:01.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:00 smithi186 ceph-mon[27599]: pgmap v108: 97 pgs: 25 active+clean, 20 creating+peering, 8 creating+activating, 44 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-24T11:21:01.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:00 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:21:01.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:00 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:21:01.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:00 smithi186 ceph-mon[27599]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T11:21:01.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:00 smithi184 ceph-mon[25816]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi184=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:21:01.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:00 smithi184 ceph-mon[25816]: Saving service alertmanager spec with placement smithi184=a;count:1 2024-03-24T11:21:01.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:00 smithi184 ceph-mon[25816]: pgmap v108: 97 pgs: 25 active+clean, 20 creating+peering, 8 creating+activating, 44 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-24T11:21:01.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:00 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:21:01.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:00 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T11:21:01.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:00 smithi184 ceph-mon[25816]: osdmap e27: 3 total, 3 up, 3 in 2024-03-24T11:21:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:01 smithi189 ceph-mon[27694]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T11:21:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:01 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:01 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:01 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.206 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled grafana update... 2024-03-24T11:21:02.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:01 smithi186 ceph-mon[27599]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T11:21:02.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:01 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:01 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:01 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:01 smithi184 ceph-mon[25816]: osdmap e28: 3 total, 3 up, 3 in 2024-03-24T11:21:02.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:01 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:01 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:01 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T11:21:02.826 DEBUG:teuthology.orchestra.run.smithi186:grafana.a> sudo journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T11:21:02.830 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:21:02.830 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:21:03.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: pgmap v111: 129 pgs: 56 active+clean, 20 creating+peering, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:21:03.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:21:03.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:02 smithi189 ceph-mon[27694]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T11:21:03.121 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: pgmap v111: 129 pgs: 56 active+clean, 20 creating+peering, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:21:03.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-24T11:21:03.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:03.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:21:03.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:21:03.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:02 smithi184 ceph-mon[25816]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T11:21:03.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: pgmap v111: 129 pgs: 56 active+clean, 20 creating+peering, 53 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 767 B/s wr, 1 op/s 2024-03-24T11:21:03.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:21:03.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T11:21:03.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: osdmap e29: 3 total, 3 up, 3 in 2024-03-24T11:21:03.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1715675533' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2997706390' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T11:21:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:02 smithi186 ceph-mon[27599]: osdmap e30: 3 total, 3 up, 3 in 2024-03-24T11:21:04.414 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:04 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.415 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:04 smithi184 ceph-mon[25816]: Deploying daemon prometheus.a on smithi186 2024-03-24T11:21:04.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:04 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:04 smithi186 ceph-mon[27599]: Deploying daemon prometheus.a on smithi186 2024-03-24T11:21:04.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:04 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:04.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:04 smithi189 ceph-mon[27694]: Deploying daemon prometheus.a on smithi186 2024-03-24T11:21:04.919 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2024-03-24T11:21:04.919 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQCgDABm0C+zNhAANhqZ/6XFhDHRIB4FRWU3Hg== 2024-03-24T11:21:05.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:05 smithi184 ceph-mon[25816]: pgmap v114: 129 pgs: 106 active+clean, 13 creating+peering, 10 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-24T11:21:05.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:05 smithi184 ceph-mon[25816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:21:05.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:05 smithi184 ceph-mon[25816]: Cluster is now healthy 2024-03-24T11:21:05.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:05 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3116277738' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:05.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:05 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3116277738' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:05.511 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:21:05.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:21:05.511 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:21:05.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:05 smithi186 ceph-mon[27599]: pgmap v114: 129 pgs: 106 active+clean, 13 creating+peering, 10 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-24T11:21:05.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:05 smithi186 ceph-mon[27599]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:21:05.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:05 smithi186 ceph-mon[27599]: Cluster is now healthy 2024-03-24T11:21:05.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3116277738' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:05.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:05 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3116277738' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:05.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:21:05.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:05 smithi189 ceph-mon[27694]: pgmap v114: 129 pgs: 106 active+clean, 13 creating+peering, 10 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-24T11:21:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:05 smithi189 ceph-mon[27694]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:21:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:05 smithi189 ceph-mon[27694]: Cluster is now healthy 2024-03-24T11:21:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:05 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3116277738' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:05.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:05 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3116277738' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:05.805 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T11:21:07.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:07 smithi189 ceph-mon[27694]: pgmap v115: 129 pgs: 106 active+clean, 13 creating+peering, 10 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.0 KiB/s rd, 690 B/s wr, 2 op/s 2024-03-24T11:21:07.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:07 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:07 smithi184 ceph-mon[25816]: pgmap v115: 129 pgs: 106 active+clean, 13 creating+peering, 10 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.0 KiB/s rd, 690 B/s wr, 2 op/s 2024-03-24T11:21:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:07 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:07 smithi186 ceph-mon[27599]: pgmap v115: 129 pgs: 106 active+clean, 13 creating+peering, 10 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.0 KiB/s rd, 690 B/s wr, 2 op/s 2024-03-24T11:21:07.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:07.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:07 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.203 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:09 smithi189 ceph-mon[27694]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 162 op/s 2024-03-24T11:21:09.203 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:09 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.204 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:09 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.204 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:09 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:09 smithi184 ceph-mon[25816]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 162 op/s 2024-03-24T11:21:09.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:09 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:09 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:09 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:09 smithi186 ceph-mon[27599]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 162 op/s 2024-03-24T11:21:09.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:09 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:09 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:09.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:09 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:11.419 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:11 smithi186 ceph-mon[27599]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-24T11:21:11.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:11 smithi184 ceph-mon[25816]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-24T11:21:11.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:11 smithi189 ceph-mon[27694]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-03-24T11:21:12.650 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-03-24T11:21:12.650 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQCoDABmHB2JJhAAk2N3f8Io/4RylvbO79oXSQ== 2024-03-24T11:21:13.387 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:13 smithi186 ceph-mon[27599]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 112 op/s 2024-03-24T11:21:13.388 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:13 smithi186 ceph-mon[27599]: from='client.? 172.21.15.186:0/51548557' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:13.388 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:13 smithi186 ceph-mon[27599]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:13.388 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:13 smithi186 ceph-mon[27599]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:13.415 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:21:13.415 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:21:13.415 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:21:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:13 smithi184 ceph-mon[25816]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 112 op/s 2024-03-24T11:21:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:13 smithi184 ceph-mon[25816]: from='client.? 172.21.15.186:0/51548557' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:13 smithi184 ceph-mon[25816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:13.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:13 smithi184 ceph-mon[25816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:13.459 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:21:13.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:13 smithi189 ceph-mon[27694]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 112 op/s 2024-03-24T11:21:13.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:13 smithi189 ceph-mon[27694]: from='client.? 172.21.15.186:0/51548557' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:13.491 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:13 smithi189 ceph-mon[27694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:13.492 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:13 smithi189 ceph-mon[27694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:13.710 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T11:21:15.194 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:15 smithi189 ceph-mon[27694]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 42 KiB/s rd, 3.1 KiB/s wr, 102 op/s 2024-03-24T11:21:15.400 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-mon[27599]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 42 KiB/s rd, 3.1 KiB/s wr, 102 op/s 2024-03-24T11:21:15.400 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 systemd[1]: Starting Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:21:15.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:15 smithi184 ceph-mon[25816]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 42 KiB/s rd, 3.1 KiB/s wr, 102 op/s 2024-03-24T11:21:15.582 INFO:teuthology.orchestra.run.smithi189.stdout:[client.2] 2024-03-24T11:21:15.582 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCrDABmE0ZtIhAAQDi/6JkMHJBmttAORzgbXA== 2024-03-24T11:21:15.656 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 podman[35923]: 2024-03-24 11:21:15.399874238 +0000 UTC m=+0.106044058 container create a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:15.656 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 podman[35923]: 2024-03-24 11:21:15.312343496 +0000 UTC m=+0.018513324 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:21:15.656 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 podman[35923]: 2024-03-24 11:21:15.626877223 +0000 UTC m=+0.333047043 container init a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:15.656 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 podman[35923]: 2024-03-24 11:21:15.632139017 +0000 UTC m=+0.338308839 container start a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.656Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.656Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.656Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.656Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.656Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.657Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:21:15.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.658Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.658Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.658Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.661Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.661Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.719µs 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.661Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.661Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.661Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.334µs wal_replay_duration=219.782µs wbl_replay_duration=91ns total_replay_duration=255.649µs 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.662Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.662Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.662Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.679Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.204882ms db_storage=731ns remote_storage=1.128µs web_handler=456ns query_engine=517ns scrape=560.982µs scrape_sd=73.468µs notify=809ns notify_sd=1.541µs rules=16.375931ms tracing=23.373µs 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.679Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:21:15.679Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:21:15.923 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 bash[35923]: a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be 2024-03-24T11:21:15.924 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:21:15 smithi186 systemd[1]: Started Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:21:16.183 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:21:16.183 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-24T11:21:16.183 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T11:21:16.266 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:21:16.266 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:21:16.266 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:21:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='client.? 172.21.15.189:0/1802828787' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:16.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='client.? 172.21.15.189:0/1802828787' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='client.? 172.21.15.189:0/1802828787' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' 2024-03-24T11:21:16.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:21:16.694 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:16.898 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ignoring --setuser ceph since I am not root 2024-03-24T11:21:16.899 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:16 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ignoring --setgroup ceph since I am not root 2024-03-24T11:21:17.009 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: ignoring --setuser ceph since I am not root 2024-03-24T11:21:17.009 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: ignoring --setgroup ceph since I am not root 2024-03-24T11:21:17.009 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:16 smithi186 ceph-mgr[29328]: -- 172.21.15.186:0/2702858017 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560d7923c1a0 con 0x560d79212c00 2024-03-24T11:21:17.115 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: ignoring --setuser ceph since I am not root 2024-03-24T11:21:17.115 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: ignoring --setgroup ceph since I am not root 2024-03-24T11:21:17.116 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:16 smithi189 ceph-mgr[30397]: -- 172.21.15.189:0/3028667959 <== mon.0 v2:172.21.15.184:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5647aa89e1a0 con 0x5647aa875000 2024-03-24T11:21:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:17 smithi184 ceph-mon[25816]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-03-24T11:21:17.421 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:17 smithi184 ceph-mon[25816]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:21:17.422 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:17 smithi184 ceph-mon[25816]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T11:21:17.422 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:17 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:17.392+0000 7f2e21aa6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:21:17.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:17 smithi189 ceph-mon[27694]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-03-24T11:21:17.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:17 smithi189 ceph-mon[27694]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:21:17.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:17 smithi189 ceph-mon[27694]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T11:21:17.616 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:17 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:17.349+0000 7f1ef98f5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:21:17.616 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:17 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:17.442+0000 7f1ef98f5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:21:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:17 smithi186 ceph-mon[27599]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.8 KiB/s wr, 92 op/s 2024-03-24T11:21:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:17 smithi186 ceph-mon[27599]: from='mgr.14150 172.21.15.184:0/1976048591' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:21:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:17 smithi186 ceph-mon[27599]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-24T11:21:17.792 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:17 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:17.361+0000 7f44f7283200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T11:21:17.792 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:17 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:17.458+0000 7f44f7283200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:21:17.812 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:17 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:17.491+0000 7f2e21aa6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T11:21:18.365 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.078+0000 7f1ef98f5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:21:18.366 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.171+0000 7f1ef98f5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:21:18.366 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.313+0000 7f1ef98f5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:21:18.407 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:18 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:18.163+0000 7f2e21aa6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:21:18.407 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:18 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:18.257+0000 7f2e21aa6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:21:18.433 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.101+0000 7f44f7283200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T11:21:18.433 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.195+0000 7f44f7283200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T11:21:18.433 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.337+0000 7f44f7283200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:21:18.433 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.433+0000 7f44f7283200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:21:18.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:18 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:18.406+0000 7f2e21aa6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T11:21:18.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:18 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:18.530+0000 7f2e21aa6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:21:18.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:18 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:18.630+0000 7f2e21aa6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:21:18.751 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.407+0000 7f1ef98f5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T11:21:18.751 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.489+0000 7f1ef98f5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:21:18.751 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.750+0000 7f1ef98f5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:21:18.781 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.516+0000 7f44f7283200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T11:21:18.782 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.781+0000 7f44f7283200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:21:19.022 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:18 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:18.908+0000 7f2e21aa6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T11:21:19.040 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.859+0000 7f1ef98f5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:21:19.040 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:18 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:18.945+0000 7f1ef98f5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:21:19.040 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:19 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:19.039+0000 7f1ef98f5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:21:19.041 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.891+0000 7f44f7283200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:21:19.042 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:18.979+0000 7f44f7283200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:21:19.365 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:19 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:19.225+0000 7f1ef98f5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:21:19.399 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:19 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:19.021+0000 7f2e21aa6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T11:21:19.399 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:19 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:19.113+0000 7f2e21aa6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T11:21:19.399 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:19 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:19.213+0000 7f2e21aa6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:21:19.399 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:19 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:19.398+0000 7f2e21aa6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:21:19.521 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:21:19.541 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:19 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:19.075+0000 7f44f7283200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T11:21:19.542 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:19 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:19.252+0000 7f44f7283200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T11:21:19.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:19 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2634697047' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:21:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:19 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2634697047' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:21:20.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:19 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2634697047' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:21:20.206 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":1077452669},{"type":"v1","addr":"172.21.15.184:6801","nonce":1077452669}]},"active_addr":"172.21.15.184:6801/1077452669","active_change":"2024-03-24T11:18:10.276683+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24122,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.184:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":4228244946}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":3001138771}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":2771148265}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":2648858132}]}]} 2024-03-24T11:21:20.212 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:21:20.213 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:21:20.213 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:21:20.456 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:20.541 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:20 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:20.237+0000 7f44f7283200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:21:20.542 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:20 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:20.369+0000 7f44f7283200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:21:20.543 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:20.396+0000 7f2e21aa6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:21:20.615 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:20 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:20.214+0000 7f1ef98f5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T11:21:20.616 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:20 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:20.348+0000 7f1ef98f5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:21:20.880 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:20.542+0000 7f2e21aa6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T11:21:20.969 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:20 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:20.683+0000 7f1ef98f5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:21:20.969 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:20 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:20.773+0000 7f1ef98f5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:21:20.969 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:20 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:20.968+0000 7f1ef98f5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:21:21.002 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:20 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:20.710+0000 7f44f7283200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:21:21.002 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:20 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:20.804+0000 7f44f7283200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:21:21.002 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:21 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:21.001+0000 7f44f7283200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:21:21.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:20.904+0000 7f2e21aa6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T11:21:21.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:21.002+0000 7f2e21aa6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T11:21:21.291 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:21 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:21.199+0000 7f44f7283200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:21:21.365 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:21 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:21.154+0000 7f1ef98f5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:21:21.600 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:21.199+0000 7f2e21aa6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:21:21.601 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:21.387+0000 7f2e21aa6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T11:21:21.700 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:21 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:21.437+0000 7f1ef98f5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:21:21.700 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:21 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:21.699+0000 7f1ef98f5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:21:21.792 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:21 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:21.509+0000 7f44f7283200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:21:21.866 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:21.685+0000 7f2e21aa6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T11:21:22.115 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:21 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:21.937+0000 7f1ef98f5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:21:22.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:21.962+0000 7f2e21aa6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:21:22.180 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:21 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:21.794+0000 7f44f7283200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T11:21:22.180 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:22 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:22.040+0000 7f44f7283200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:21:22.293 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:21:22.293 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":30,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:17:13.767382+0000","modified":"2024-03-24T11:21:02.942490+0000","last_up_change":"2024-03-24T11:20:42.067203+0000","last_in_change":"2024-03-24T11:20:21.821174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:20:42.614362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:20:54.286818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:20:56.233329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:20:58.192796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:21:00.232373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb30c42f-c967-489d-95fd-3abe16af1b45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6803","nonce":3760044605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6805","nonce":3760044605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6809","nonce":3760044605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6807","nonce":3760044605}]},"public_addr":"172.21.15.184:6803/3760044605","cluster_addr":"172.21.15.184:6805/3760044605","heartbeat_back_addr":"172.21.15.184:6809/3760044605","heartbeat_front_addr":"172.21.15.184:6807/3760044605","state":["exists","up"]},{"osd":1,"uuid":"83c102f5-1b96-4121-b771-85a4eb8fc5eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6801","nonce":682526296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6803","nonce":682526296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6807","nonce":682526296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6805","nonce":682526296}]},"public_addr":"172.21.15.186:6801/682526296","cluster_addr":"172.21.15.186:6803/682526296","heartbeat_back_addr":"172.21.15.186:6807/682526296","heartbeat_front_addr":"172.21.15.186:6805/682526296","state":["exists","up"]},{"osd":2,"uuid":"17cd9e6d-11f8-4c70-8cab-b26eec2df1c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3420409320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3420409320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3420409320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3420409320}]},"public_addr":"172.21.15.189:6801/3420409320","cluster_addr":"172.21.15.189:6803/3420409320","heartbeat_back_addr":"172.21.15.189:6807/3420409320","heartbeat_front_addr":"172.21.15.189:6805/3420409320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:47.901976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:20:13.957580+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:20:40.084873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:0/3013664472":"2024-03-25T11:18:10.276568+0000","172.21.15.184:6800/4057035916":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/1363820209":"2024-03-25T11:17:43.481718+0000","172.21.15.184:6800/873414609":"2024-03-25T11:17:43.481718+0000","172.21.15.184:6801/4057035916":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/2401055791":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/3563561133":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/2143438781":"2024-03-25T11:18:10.276568+0000","172.21.15.184:6801/873414609":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/544235517":"2024-03-25T11:17:43.481718+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:21:22.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:22 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3120937072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:21:22.607 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:22 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3120937072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:21:22.607 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:22 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:22.210+0000 7f2e21aa6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T11:21:22.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:22 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3120937072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:21:22.918 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:21:22.918 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:21:23.115 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:22 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:22.743+0000 7f1ef98f5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:21:23.163 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:23.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:23.043+0000 7f2e21aa6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:21:23.291 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:22 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:22.859+0000 7f44f7283200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T11:21:23.615 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:23.316+0000 7f1ef98f5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:21:23.616 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:23.415+0000 7f1ef98f5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:21:23.710 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:23.433+0000 7f44f7283200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:21:23.710 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:23.533+0000 7f44f7283200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:21:23.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-mon[25816]: Standby manager daemon z restarted 2024-03-24T11:21:23.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-mon[25816]: Standby manager daemon z started 2024-03-24T11:21:23.888 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:21:23.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:21:23.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:21:23.889 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:21:23.889 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:23.659+0000 7f2e21aa6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T11:21:23.889 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:23.762+0000 7f2e21aa6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T11:21:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-mon[27599]: Standby manager daemon z restarted 2024-03-24T11:21:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-mon[27599]: Standby manager daemon z started 2024-03-24T11:21:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:21:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:21:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:21:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:23.755+0000 7f44f7283200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: [24/Mar/2024:11:21:23] ENGINE Bus STARTING 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: CherryPy Checker: 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: The Application mounted at '' has an empty config. 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: [24/Mar/2024:11:21:23] ENGINE Serving on http://:::9283 2024-03-24T11:21:24.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:21:23 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x[29305]: [24/Mar/2024:11:21:23] ENGINE Bus STARTED 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-mon[27694]: Standby manager daemon z restarted 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-mon[27694]: Standby manager daemon z started 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.189:0/3158428842' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:23.633+0000 7f1ef98f5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:21:24.116 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: [24/Mar/2024:11:21:23] ENGINE Bus STARTING 2024-03-24T11:21:24.117 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: CherryPy Checker: 2024-03-24T11:21:24.117 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: The Application mounted at '' has an empty config. 2024-03-24T11:21:24.117 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: 2024-03-24T11:21:24.117 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: [24/Mar/2024:11:21:23] ENGINE Serving on http://:::9283 2024-03-24T11:21:24.117 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:21:23 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z[30337]: [24/Mar/2024:11:21:23] ENGINE Bus STARTED 2024-03-24T11:21:24.148 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:23 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:23.986+0000 7f2e21aa6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T11:21:24.148 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:24] ENGINE Bus STARTING 2024-03-24T11:21:24.421 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: CherryPy Checker: 2024-03-24T11:21:24.422 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: The Application mounted at '' has an empty config. 2024-03-24T11:21:24.422 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: 2024-03-24T11:21:24.422 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:24] ENGINE Serving on http://:::9283 2024-03-24T11:21:24.422 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:24] ENGINE Bus STARTED 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: Standby manager daemon x restarted 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: Standby manager daemon x started 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: Active manager daemon y restarted 2024-03-24T11:21:24.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: Activating manager daemon y 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: mgrmap e19: y(active, starting, since 0.00973398s), standbys: x, z 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:21:24.923 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: Manager daemon y is now available 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:24.924 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:21:24.925 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:21:25.002 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:21:25.002 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":31,"fsid":"f43a2692-e9cf-11ee-95cd-87774f69a715","created":"2024-03-24T11:17:13.767382+0000","modified":"2024-03-24T11:21:24.000701+0000","last_up_change":"2024-03-24T11:20:42.067203+0000","last_in_change":"2024-03-24T11:20:21.821174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:20:42.614362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-24T11:20:54.286818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-24T11:20:56.233329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-24T11:20:58.192796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-24T11:21:00.232373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"eb30c42f-c967-489d-95fd-3abe16af1b45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6803","nonce":3760044605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6805","nonce":3760044605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6809","nonce":3760044605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":3760044605},{"type":"v1","addr":"172.21.15.184:6807","nonce":3760044605}]},"public_addr":"172.21.15.184:6803/3760044605","cluster_addr":"172.21.15.184:6805/3760044605","heartbeat_back_addr":"172.21.15.184:6809/3760044605","heartbeat_front_addr":"172.21.15.184:6807/3760044605","state":["exists","up"]},{"osd":1,"uuid":"83c102f5-1b96-4121-b771-85a4eb8fc5eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6801","nonce":682526296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6803","nonce":682526296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6807","nonce":682526296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":682526296},{"type":"v1","addr":"172.21.15.186:6805","nonce":682526296}]},"public_addr":"172.21.15.186:6801/682526296","cluster_addr":"172.21.15.186:6803/682526296","heartbeat_back_addr":"172.21.15.186:6807/682526296","heartbeat_front_addr":"172.21.15.186:6805/682526296","state":["exists","up"]},{"osd":2,"uuid":"17cd9e6d-11f8-4c70-8cab-b26eec2df1c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6801","nonce":3420409320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6803","nonce":3420409320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6807","nonce":3420409320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3420409320},{"type":"v1","addr":"172.21.15.189:6805","nonce":3420409320}]},"public_addr":"172.21.15.189:6801/3420409320","cluster_addr":"172.21.15.189:6803/3420409320","heartbeat_back_addr":"172.21.15.189:6807/3420409320","heartbeat_front_addr":"172.21.15.189:6805/3420409320","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:19:47.901976+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:20:13.957580+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:20:40.084873+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.184:0/4228244946":"2024-03-25T11:21:24.000653+0000","172.21.15.184:6800/1077452669":"2024-03-25T11:21:24.000653+0000","172.21.15.184:0/3001138771":"2024-03-25T11:21:24.000653+0000","172.21.15.184:6801/1077452669":"2024-03-25T11:21:24.000653+0000","172.21.15.184:0/2648858132":"2024-03-25T11:21:24.000653+0000","172.21.15.184:0/3013664472":"2024-03-25T11:18:10.276568+0000","172.21.15.184:6800/4057035916":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/1363820209":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/2771148265":"2024-03-25T11:21:24.000653+0000","172.21.15.184:6800/873414609":"2024-03-25T11:17:43.481718+0000","172.21.15.184:6801/4057035916":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/2401055791":"2024-03-25T11:18:10.276568+0000","172.21.15.184:0/3563561133":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/2143438781":"2024-03-25T11:18:10.276568+0000","172.21.15.184:6801/873414609":"2024-03-25T11:17:43.481718+0000","172.21.15.184:0/544235517":"2024-03-25T11:17:43.481718+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:21:25.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: Standby manager daemon x restarted 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: Standby manager daemon x started 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: Active manager daemon y restarted 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: Activating manager daemon y 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: mgrmap e19: y(active, starting, since 0.00973398s), standbys: x, z 2024-03-24T11:21:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: Manager daemon y is now available 2024-03-24T11:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:25.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:21:25.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: Standby manager daemon x restarted 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: Standby manager daemon x started 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.? 172.21.15.186:0/583048154' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: Active manager daemon y restarted 2024-03-24T11:21:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: Activating manager daemon y 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: osdmap e31: 3 total, 3 up, 3 in 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: mgrmap e19: y(active, starting, since 0.00973398s), standbys: x, z 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:21:25.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: Manager daemon y is now available 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi184", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]': finished 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:25.118 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:21:25.119 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T11:21:25.709 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:21:25.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:21:25.710 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:21:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:25.984 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:25.985 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:26.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:21:26.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:21:26.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:21:26.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: [24/Mar/2024:11:21:24] ENGINE Bus STARTING 2024-03-24T11:21:26.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: [24/Mar/2024:11:21:24] ENGINE Serving on https://172.21.15.184:7151 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: [24/Mar/2024:11:21:24] ENGINE Serving on http://172.21.15.184:8765 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: [24/Mar/2024:11:21:24] ENGINE Bus STARTED 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3414477552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: mgrmap e20: y(active, since 1.01644s), standbys: x, z 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-03-24T11:21:26.071 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:21:26.071 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: [24/Mar/2024:11:21:24] ENGINE Bus STARTING 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: [24/Mar/2024:11:21:24] ENGINE Serving on https://172.21.15.184:7151 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: [24/Mar/2024:11:21:24] ENGINE Serving on http://172.21.15.184:8765 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: [24/Mar/2024:11:21:24] ENGINE Bus STARTED 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3414477552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: mgrmap e20: y(active, since 1.01644s), standbys: x, z 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.072 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.073 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: Updating smithi184:/etc/ceph/ceph.conf 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: [24/Mar/2024:11:21:24] ENGINE Bus STARTING 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: [24/Mar/2024:11:21:24] ENGINE Serving on https://172.21.15.184:7151 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: [24/Mar/2024:11:21:24] ENGINE Serving on http://172.21.15.184:8765 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: [24/Mar/2024:11:21:24] ENGINE Bus STARTED 2024-03-24T11:21:26.111 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3414477552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: mgrmap e20: y(active, since 1.01644s), standbys: x, z 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.112 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi184", "caps": []}]: dispatch 2024-03-24T11:21:26.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:26.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:26.879 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]: dispatch 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.880 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:26 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:26.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:26.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:26.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:26.957 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]: dispatch 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:26.958 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:26 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Updating smithi184:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Updating smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Updating smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Updating smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:21:27.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi186", "caps": []}]: dispatch 2024-03-24T11:21:27.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:26 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:27.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T11:21:27.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:21:27.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:27.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:27 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:21:27.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:27.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T11:21:27.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:21:27.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:27 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:27.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:27.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:27 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi189", "caps": []}]: dispatch 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.117 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:29.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:29.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.181 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.181 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:29.181 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:29.181 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:29.182 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:30.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:29 smithi184 ceph-mon[25816]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-24T11:21:30.172 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:29 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:30.291 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:29 smithi186 ceph-mon[27599]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-24T11:21:30.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:29 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:30.304 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:29 smithi189 ceph-mon[27694]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-24T11:21:30.304 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:29 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:30 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:21:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:31.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:31 smithi184 ceph-mon[25816]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:31.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:31 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:31 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.506 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:31 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.507 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:31 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.536 INFO:teuthology.orchestra.run.smithi184.stdout:34359738391 2024-03-24T11:21:31.536 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:21:31.537 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:31 smithi189 ceph-mon[27694]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:31.537 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:31 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.537 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:31 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.538 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:31 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.538 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:31 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:31 smithi186 ceph-mon[27599]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:31.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:31 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:31 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.541 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:31 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:31.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:31 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:32.651 INFO:teuthology.orchestra.run.smithi184.stdout:55834574865 2024-03-24T11:21:32.651 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:21:33.026 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:33.279 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:21:33.279 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.279 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.279 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.280 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.280 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.280 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.280 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.280 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:33 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:33 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:33.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:33 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.063 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:34.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.526 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: Deploying daemon alertmanager.a on smithi184 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.527 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:34 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: Deploying daemon alertmanager.a on smithi184 2024-03-24T11:21:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:34 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:21:34.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: Deploying daemon alertmanager.a on smithi184 2024-03-24T11:21:34.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:34.617 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:34 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:35.758 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:35 smithi184 ceph-mon[25816]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:21:35.759 INFO:teuthology.orchestra.run.smithi184.stdout:77309411339 2024-03-24T11:21:35.760 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:21:36.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:35 smithi186 ceph-mon[27599]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:21:36.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:35 smithi189 ceph-mon[27694]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T11:21:38.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:37 smithi186 ceph-mon[27599]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:21:38.080 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:37 smithi184 ceph-mon[25816]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:21:38.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:37 smithi189 ceph-mon[27694]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T11:21:38.288 INFO:teuthology.orchestra.run.smithi184.stdout:34359738392 2024-03-24T11:21:38.859 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:39.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:38 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/3579120174' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:21:39.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:38 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/3579120174' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:21:39.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:38 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/3579120174' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:21:39.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:39 smithi184 ceph-mon[25816]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:39.896 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:39 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:40.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:39 smithi186 ceph-mon[27599]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:40.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:39 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:40.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738391 got 34359738392 for osd.0 2024-03-24T11:21:40.106 DEBUG:teuthology.parallel:result is None 2024-03-24T11:21:40.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:39 smithi189 ceph-mon[27694]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:40.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:39 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:40.921 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:40 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:21:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:41.688 INFO:teuthology.orchestra.run.smithi184.stdout:55834574866 2024-03-24T11:21:41.837 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:41 smithi184 ceph-mon[25816]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:41.837 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:41 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1650972543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:21:42.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:41 smithi186 ceph-mon[27599]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:42.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:41 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1650972543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:21:42.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:41 smithi189 ceph-mon[27694]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:42.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:41 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1650972543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:21:42.700 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574866 for osd.1 2024-03-24T11:21:42.700 DEBUG:teuthology.parallel:result is None 2024-03-24T11:21:44.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:43 smithi186 ceph-mon[27599]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:44.060 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:43 smithi184 ceph-mon[25816]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:44.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:43 smithi189 ceph-mon[27694]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T11:21:45.252 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:45 smithi184 systemd[1]: Starting Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:21:45.581 INFO:teuthology.orchestra.run.smithi184.stdout:77309411341 2024-03-24T11:21:45.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:45 smithi184 ceph-mon[25816]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:45.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:45 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/2466233239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:21:45.921 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:45 smithi184 podman[48942]: 2024-03-24 11:21:45.502120217 +0000 UTC m=+0.101250824 volume create 29b33e1ca44c70017e8ce183dffb2e5a8429f353c5748fc4a045c48da5dbe411 2024-03-24T11:21:45.921 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:45 smithi184 podman[48942]: 2024-03-24 11:21:45.418716743 +0000 UTC m=+0.017847343 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-24T11:21:45.922 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:45 smithi184 podman[48942]: 2024-03-24 11:21:45.569148839 +0000 UTC m=+0.168279444 container create aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:46.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:45 smithi186 ceph-mon[27599]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:46.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:45 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/2466233239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:21:46.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:45 smithi189 ceph-mon[27694]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:46.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:45 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/2466233239' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 podman[48942]: 2024-03-24 11:21:46.097803231 +0000 UTC m=+0.696933838 container init aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 podman[48942]: 2024-03-24 11:21:46.102735252 +0000 UTC m=+0.701865850 container start aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.115Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.115Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.116Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.184 port=9094 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.117Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.140Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:21:46.368 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.140Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:21:46.369 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.143Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:21:46.369 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:46.143Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:21:46.369 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 bash[48942]: aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 2024-03-24T11:21:46.369 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:46 smithi184 systemd[1]: Started Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:21:46.660 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:46] ENGINE Bus STOPPING 2024-03-24T11:21:46.660 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:21:46.660 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:46] ENGINE Bus STOPPED 2024-03-24T11:21:46.660 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:46] ENGINE Bus STARTING 2024-03-24T11:21:46.660 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:46] ENGINE Serving on http://:::9283 2024-03-24T11:21:46.660 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:46 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:21:46] ENGINE Bus STARTED 2024-03-24T11:21:46.707 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411341 for osd.2 2024-03-24T11:21:46.707 DEBUG:teuthology.parallel:result is None 2024-03-24T11:21:46.707 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:21:46.707 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:21:47.232 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:21:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:21:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:47 smithi186 ceph-mon[27599]: Deploying daemon grafana.a on smithi186 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:47 smithi189 ceph-mon[27694]: Deploying daemon grafana.a on smithi186 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T11:21:47.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:21:47.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:21:47.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:47.673 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:47 smithi184 ceph-mon[25816]: Deploying daemon grafana.a on smithi186 2024-03-24T11:21:48.373 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:48 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:48.117Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000301503s 2024-03-24T11:21:50.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:49 smithi186 ceph-mon[27599]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:50.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:49 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:50.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:49 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:50.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:49 smithi189 ceph-mon[27694]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:50.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:49 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:50.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:49 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:49 smithi184 ceph-mon[25816]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:49 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:50.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:49 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:50.794 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:21:50 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:21:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:21:51.429 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:21:51.432 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2024-03-24T11:21:51.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:51 smithi189 ceph-mon[27694]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:51.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:51 smithi184 ceph-mon[25816]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:52.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:51 smithi186 ceph-mon[27599]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:52.266 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-03-24T11:21:50.021508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008153+0000","last_change":"2024-03-24T11:21:01.958959+0000","last_active":"2024-03-24T11:21:24.008153+0000","last_peered":"2024-03-24T11:21:24.008153+0000","last_clean":"2024-03-24T11:21:24.008153+0000","last_became_active":"2024-03-24T11:21:01.958420+0000","last_became_peered":"2024-03-24T11:21:01.958420+0000","last_unstale":"2024-03-24T11:21:24.008153+0000","last_undegraded":"2024-03-24T11:21:24.008153+0000","last_fullsized":"2024-03-24T11:21:24.008153+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:37:14.236555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007790+0000","last_change":"2024-03-24T11:20:59.942774+0000","last_active":"2024-03-24T11:21:24.007790+0000","last_peered":"2024-03-24T11:21:24.007790+0000","last_clean":"2024-03-24T11:21:24.007790+0000","last_became_active":"2024-03-24T11:20:59.941212+0000","last_became_peered":"2024-03-24T11:20:59.941212+0000","last_unstale":"2024-03-24T11:21:24.007790+0000","last_undegraded":"2024-03-24T11:21:24.007790+0000","last_fullsized":"2024-03-24T11:21:24.007790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:14:35.912122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008901+0000","last_change":"2024-03-24T11:20:55.923231+0000","last_active":"2024-03-24T11:21:24.008901+0000","last_peered":"2024-03-24T11:21:24.008901+0000","last_clean":"2024-03-24T11:21:24.008901+0000","last_became_active":"2024-03-24T11:20:55.922900+0000","last_became_peered":"2024-03-24T11:20:55.922900+0000","last_unstale":"2024-03-24T11:21:24.008901+0000","last_undegraded":"2024-03-24T11:21:24.008901+0000","last_fullsized":"2024-03-24T11:21:24.008901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:18.794423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009094+0000","last_change":"2024-03-24T11:20:57.934823+0000","last_active":"2024-03-24T11:21:24.009094+0000","last_peered":"2024-03-24T11:21:24.009094+0000","last_clean":"2024-03-24T11:21:24.009094+0000","last_became_active":"2024-03-24T11:20:57.934393+0000","last_became_peered":"2024-03-24T11:20:57.934393+0000","last_unstale":"2024-03-24T11:21:24.009094+0000","last_undegraded":"2024-03-24T11:21:24.009094+0000","last_fullsized":"2024-03-24T11:21:24.009094+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:23.379426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006465+0000","last_change":"2024-03-24T11:21:01.959790+0000","last_active":"2024-03-24T11:21:24.006465+0000","last_peered":"2024-03-24T11:21:24.006465+0000","last_clean":"2024-03-24T11:21:24.006465+0000","last_became_active":"2024-03-24T11:21:01.959476+0000","last_became_peered":"2024-03-24T11:21:01.959476+0000","last_unstale":"2024-03-24T11:21:24.006465+0000","last_undegraded":"2024-03-24T11:21:24.006465+0000","last_fullsized":"2024-03-24T11:21:24.006465+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:49:03.839785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008586+0000","last_change":"2024-03-24T11:20:59.944547+0000","last_active":"2024-03-24T11:21:24.008586+0000","last_peered":"2024-03-24T11:21:24.008586+0000","last_clean":"2024-03-24T11:21:24.008586+0000","last_became_active":"2024-03-24T11:20:59.943831+0000","last_became_peered":"2024-03-24T11:20:59.943831+0000","last_unstale":"2024-03-24T11:21:24.008586+0000","last_undegraded":"2024-03-24T11:21:24.008586+0000","last_fullsized":"2024-03-24T11:21:24.008586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:32:57.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009314+0000","last_change":"2024-03-24T11:20:55.925563+0000","last_active":"2024-03-24T11:21:24.009314+0000","last_peered":"2024-03-24T11:21:24.009314+0000","last_clean":"2024-03-24T11:21:24.009314+0000","last_became_active":"2024-03-24T11:20:55.924799+0000","last_became_peered":"2024-03-24T11:20:55.924799+0000","last_unstale":"2024-03-24T11:21:24.009314+0000","last_undegraded":"2024-03-24T11:21:24.009314+0000","last_fullsized":"2024-03-24T11:21:24.009314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:07.051785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008540+0000","last_change":"2024-03-24T11:20:57.934187+0000","last_active":"2024-03-24T11:21:24.008540+0000","last_peered":"2024-03-24T11:21:24.008540+0000","last_clean":"2024-03-24T11:21:24.008540+0000","last_became_active":"2024-03-24T11:20:57.933902+0000","last_became_peered":"2024-03-24T11:20:57.933902+0000","last_unstale":"2024-03-24T11:21:24.008540+0000","last_undegraded":"2024-03-24T11:21:24.008540+0000","last_fullsized":"2024-03-24T11:21:24.008540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:27.035959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008115+0000","last_change":"2024-03-24T11:21:01.953650+0000","last_active":"2024-03-24T11:21:24.008115+0000","last_peered":"2024-03-24T11:21:24.008115+0000","last_clean":"2024-03-24T11:21:24.008115+0000","last_became_active":"2024-03-24T11:21:01.953361+0000","last_became_peered":"2024-03-24T11:21:01.953361+0000","last_unstale":"2024-03-24T11:21:24.008115+0000","last_undegraded":"2024-03-24T11:21:24.008115+0000","last_fullsized":"2024-03-24T11:21:24.008115+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:57:37.168749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010023+0000","last_change":"2024-03-24T11:20:59.944256+0000","last_active":"2024-03-24T11:21:24.010023+0000","last_peered":"2024-03-24T11:21:24.010023+0000","last_clean":"2024-03-24T11:21:24.010023+0000","last_became_active":"2024-03-24T11:20:59.943832+0000","last_became_peered":"2024-03-24T11:20:59.943832+0000","last_unstale":"2024-03-24T11:21:24.010023+0000","last_undegraded":"2024-03-24T11:21:24.010023+0000","last_fullsized":"2024-03-24T11:21:24.010023+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:36:01.606131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011335+0000","last_change":"2024-03-24T11:20:55.931270+0000","last_active":"2024-03-24T11:21:24.011335+0000","last_peered":"2024-03-24T11:21:24.011335+0000","last_clean":"2024-03-24T11:21:24.011335+0000","last_became_active":"2024-03-24T11:20:55.929673+0000","last_became_peered":"2024-03-24T11:20:55.929673+0000","last_unstale":"2024-03-24T11:21:24.011335+0000","last_undegraded":"2024-03-24T11:21:24.011335+0000","last_fullsized":"2024-03-24T11:21:24.011335+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:22.177943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009791+0000","last_change":"2024-03-24T11:20:57.935522+0000","last_active":"2024-03-24T11:21:24.009791+0000","last_peered":"2024-03-24T11:21:24.009791+0000","last_clean":"2024-03-24T11:21:24.009791+0000","last_became_active":"2024-03-24T11:20:57.935122+0000","last_became_peered":"2024-03-24T11:20:57.935122+0000","last_unstale":"2024-03-24T11:21:24.009791+0000","last_undegraded":"2024-03-24T11:21:24.009791+0000","last_fullsized":"2024-03-24T11:21:24.009791+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:47:18.396876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009156+0000","last_change":"2024-03-24T11:21:01.956657+0000","last_active":"2024-03-24T11:21:24.009156+0000","last_peered":"2024-03-24T11:21:24.009156+0000","last_clean":"2024-03-24T11:21:24.009156+0000","last_became_active":"2024-03-24T11:21:01.956017+0000","last_became_peered":"2024-03-24T11:21:01.956017+0000","last_unstale":"2024-03-24T11:21:24.009156+0000","last_undegraded":"2024-03-24T11:21:24.009156+0000","last_fullsized":"2024-03-24T11:21:24.009156+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:30:53.208222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008835+0000","last_change":"2024-03-24T11:20:59.941189+0000","last_active":"2024-03-24T11:21:24.008835+0000","last_peered":"2024-03-24T11:21:24.008835+0000","last_clean":"2024-03-24T11:21:24.008835+0000","last_became_active":"2024-03-24T11:20:59.940825+0000","last_became_peered":"2024-03-24T11:20:59.940825+0000","last_unstale":"2024-03-24T11:21:24.008835+0000","last_undegraded":"2024-03-24T11:21:24.008835+0000","last_fullsized":"2024-03-24T11:21:24.008835+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:44:10.472366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009737+0000","last_change":"2024-03-24T11:20:55.929779+0000","last_active":"2024-03-24T11:21:24.009737+0000","last_peered":"2024-03-24T11:21:24.009737+0000","last_clean":"2024-03-24T11:21:24.009737+0000","last_became_active":"2024-03-24T11:20:55.929445+0000","last_became_peered":"2024-03-24T11:20:55.929445+0000","last_unstale":"2024-03-24T11:21:24.009737+0000","last_undegraded":"2024-03-24T11:21:24.009737+0000","last_fullsized":"2024-03-24T11:21:24.009737+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:00.920661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011203+0000","last_change":"2024-03-24T11:20:57.931822+0000","last_active":"2024-03-24T11:21:24.011203+0000","last_peered":"2024-03-24T11:21:24.011203+0000","last_clean":"2024-03-24T11:21:24.011203+0000","last_became_active":"2024-03-24T11:20:57.931399+0000","last_became_peered":"2024-03-24T11:20:57.931399+0000","last_unstale":"2024-03-24T11:21:24.011203+0000","last_undegraded":"2024-03-24T11:21:24.011203+0000","last_fullsized":"2024-03-24T11:21:24.011203+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:19:08.715267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009413+0000","last_change":"2024-03-24T11:21:01.957599+0000","last_active":"2024-03-24T11:21:24.009413+0000","last_peered":"2024-03-24T11:21:24.009413+0000","last_clean":"2024-03-24T11:21:24.009413+0000","last_became_active":"2024-03-24T11:21:01.956921+0000","last_became_peered":"2024-03-24T11:21:01.956921+0000","last_unstale":"2024-03-24T11:21:24.009413+0000","last_undegraded":"2024-03-24T11:21:24.009413+0000","last_fullsized":"2024-03-24T11:21:24.009413+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:51:02.097620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010110+0000","last_change":"2024-03-24T11:20:59.947557+0000","last_active":"2024-03-24T11:21:24.010110+0000","last_peered":"2024-03-24T11:21:24.010110+0000","last_clean":"2024-03-24T11:21:24.010110+0000","last_became_active":"2024-03-24T11:20:59.947251+0000","last_became_peered":"2024-03-24T11:20:59.947251+0000","last_unstale":"2024-03-24T11:21:24.010110+0000","last_undegraded":"2024-03-24T11:21:24.010110+0000","last_fullsized":"2024-03-24T11:21:24.010110+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:08:17.836024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010221+0000","last_change":"2024-03-24T11:20:55.930046+0000","last_active":"2024-03-24T11:21:24.010221+0000","last_peered":"2024-03-24T11:21:24.010221+0000","last_clean":"2024-03-24T11:21:24.010221+0000","last_became_active":"2024-03-24T11:20:55.929529+0000","last_became_peered":"2024-03-24T11:20:55.929529+0000","last_unstale":"2024-03-24T11:21:24.010221+0000","last_undegraded":"2024-03-24T11:21:24.010221+0000","last_fullsized":"2024-03-24T11:21:24.010221+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:29:58.266487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009073+0000","last_change":"2024-03-24T11:20:57.928352+0000","last_active":"2024-03-24T11:21:24.009073+0000","last_peered":"2024-03-24T11:21:24.009073+0000","last_clean":"2024-03-24T11:21:24.009073+0000","last_became_active":"2024-03-24T11:20:57.927984+0000","last_became_peered":"2024-03-24T11:20:57.927984+0000","last_unstale":"2024-03-24T11:21:24.009073+0000","last_undegraded":"2024-03-24T11:21:24.009073+0000","last_fullsized":"2024-03-24T11:21:24.009073+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:47.835375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007419+0000","last_change":"2024-03-24T11:21:01.962260+0000","last_active":"2024-03-24T11:21:24.007419+0000","last_peered":"2024-03-24T11:21:24.007419+0000","last_clean":"2024-03-24T11:21:24.007419+0000","last_became_active":"2024-03-24T11:21:01.961596+0000","last_became_peered":"2024-03-24T11:21:01.961596+0000","last_unstale":"2024-03-24T11:21:24.007419+0000","last_undegraded":"2024-03-24T11:21:24.007419+0000","last_fullsized":"2024-03-24T11:21:24.007419+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:41:53.989614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009574+0000","last_change":"2024-03-24T11:20:59.946232+0000","last_active":"2024-03-24T11:21:24.009574+0000","last_peered":"2024-03-24T11:21:24.009574+0000","last_clean":"2024-03-24T11:21:24.009574+0000","last_became_active":"2024-03-24T11:20:59.945921+0000","last_became_peered":"2024-03-24T11:20:59.945921+0000","last_unstale":"2024-03-24T11:21:24.009574+0000","last_undegraded":"2024-03-24T11:21:24.009574+0000","last_fullsized":"2024-03-24T11:21:24.009574+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:28.418393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009321+0000","last_change":"2024-03-24T11:20:55.923351+0000","last_active":"2024-03-24T11:21:24.009321+0000","last_peered":"2024-03-24T11:21:24.009321+0000","last_clean":"2024-03-24T11:21:24.009321+0000","last_became_active":"2024-03-24T11:20:55.922749+0000","last_became_peered":"2024-03-24T11:20:55.922749+0000","last_unstale":"2024-03-24T11:21:24.009321+0000","last_undegraded":"2024-03-24T11:21:24.009321+0000","last_fullsized":"2024-03-24T11:21:24.009321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:35:13.612167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009214+0000","last_change":"2024-03-24T11:20:57.931769+0000","last_active":"2024-03-24T11:21:24.009214+0000","last_peered":"2024-03-24T11:21:24.009214+0000","last_clean":"2024-03-24T11:21:24.009214+0000","last_became_active":"2024-03-24T11:20:57.931445+0000","last_became_peered":"2024-03-24T11:20:57.931445+0000","last_unstale":"2024-03-24T11:21:24.009214+0000","last_undegraded":"2024-03-24T11:21:24.009214+0000","last_fullsized":"2024-03-24T11:21:24.009214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:57.613138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007780+0000","last_change":"2024-03-24T11:21:01.961129+0000","last_active":"2024-03-24T11:21:24.007780+0000","last_peered":"2024-03-24T11:21:24.007780+0000","last_clean":"2024-03-24T11:21:24.007780+0000","last_became_active":"2024-03-24T11:21:01.960598+0000","last_became_peered":"2024-03-24T11:21:01.960598+0000","last_unstale":"2024-03-24T11:21:24.007780+0000","last_undegraded":"2024-03-24T11:21:24.007780+0000","last_fullsized":"2024-03-24T11:21:24.007780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:28:54.547856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977570+0000","last_change":"2024-03-24T11:20:59.948168+0000","last_active":"2024-03-24T11:21:40.977570+0000","last_peered":"2024-03-24T11:21:40.977570+0000","last_clean":"2024-03-24T11:21:40.977570+0000","last_became_active":"2024-03-24T11:20:59.947331+0000","last_became_peered":"2024-03-24T11:20:59.947331+0000","last_unstale":"2024-03-24T11:21:40.977570+0000","last_undegraded":"2024-03-24T11:21:40.977570+0000","last_fullsized":"2024-03-24T11:21:40.977570+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:16.261126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006794+0000","last_change":"2024-03-24T11:20:55.925062+0000","last_active":"2024-03-24T11:21:24.006794+0000","last_peered":"2024-03-24T11:21:24.006794+0000","last_clean":"2024-03-24T11:21:24.006794+0000","last_became_active":"2024-03-24T11:20:55.924606+0000","last_became_peered":"2024-03-24T11:20:55.924606+0000","last_unstale":"2024-03-24T11:21:24.006794+0000","last_undegraded":"2024-03-24T11:21:24.006794+0000","last_fullsized":"2024-03-24T11:21:24.006794+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:40:56.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005886+0000","last_change":"2024-03-24T11:20:57.934243+0000","last_active":"2024-03-24T11:21:24.005886+0000","last_peered":"2024-03-24T11:21:24.005886+0000","last_clean":"2024-03-24T11:21:24.005886+0000","last_became_active":"2024-03-24T11:20:57.933810+0000","last_became_peered":"2024-03-24T11:20:57.933810+0000","last_unstale":"2024-03-24T11:21:24.005886+0000","last_undegraded":"2024-03-24T11:21:24.005886+0000","last_fullsized":"2024-03-24T11:21:24.005886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:50:27.488867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009661+0000","last_change":"2024-03-24T11:21:01.958387+0000","last_active":"2024-03-24T11:21:24.009661+0000","last_peered":"2024-03-24T11:21:24.009661+0000","last_clean":"2024-03-24T11:21:24.009661+0000","last_became_active":"2024-03-24T11:21:01.957553+0000","last_became_peered":"2024-03-24T11:21:01.957553+0000","last_unstale":"2024-03-24T11:21:24.009661+0000","last_undegraded":"2024-03-24T11:21:24.009661+0000","last_fullsized":"2024-03-24T11:21:24.009661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.335166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008797+0000","last_change":"2024-03-24T11:20:59.941855+0000","last_active":"2024-03-24T11:21:24.008797+0000","last_peered":"2024-03-24T11:21:24.008797+0000","last_clean":"2024-03-24T11:21:24.008797+0000","last_became_active":"2024-03-24T11:20:59.941069+0000","last_became_peered":"2024-03-24T11:20:59.941069+0000","last_unstale":"2024-03-24T11:21:24.008797+0000","last_undegraded":"2024-03-24T11:21:24.008797+0000","last_fullsized":"2024-03-24T11:21:24.008797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:53.610769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008139+0000","last_change":"2024-03-24T11:20:55.930396+0000","last_active":"2024-03-24T11:21:24.008139+0000","last_peered":"2024-03-24T11:21:24.008139+0000","last_clean":"2024-03-24T11:21:24.008139+0000","last_became_active":"2024-03-24T11:20:55.929648+0000","last_became_peered":"2024-03-24T11:20:55.929648+0000","last_unstale":"2024-03-24T11:21:24.008139+0000","last_undegraded":"2024-03-24T11:21:24.008139+0000","last_fullsized":"2024-03-24T11:21:24.008139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:21.885318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005991+0000","last_change":"2024-03-24T11:20:57.932568+0000","last_active":"2024-03-24T11:21:24.005991+0000","last_peered":"2024-03-24T11:21:24.005991+0000","last_clean":"2024-03-24T11:21:24.005991+0000","last_became_active":"2024-03-24T11:20:57.932231+0000","last_became_peered":"2024-03-24T11:20:57.932231+0000","last_unstale":"2024-03-24T11:21:24.005991+0000","last_undegraded":"2024-03-24T11:21:24.005991+0000","last_fullsized":"2024-03-24T11:21:24.005991+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:07:44.514789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008780+0000","last_change":"2024-03-24T11:21:01.958125+0000","last_active":"2024-03-24T11:21:24.008780+0000","last_peered":"2024-03-24T11:21:24.008780+0000","last_clean":"2024-03-24T11:21:24.008780+0000","last_became_active":"2024-03-24T11:21:01.956305+0000","last_became_peered":"2024-03-24T11:21:01.956305+0000","last_unstale":"2024-03-24T11:21:24.008780+0000","last_undegraded":"2024-03-24T11:21:24.008780+0000","last_fullsized":"2024-03-24T11:21:24.008780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:18:54.892727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005886+0000","last_change":"2024-03-24T11:20:59.941565+0000","last_active":"2024-03-24T11:21:24.005886+0000","last_peered":"2024-03-24T11:21:24.005886+0000","last_clean":"2024-03-24T11:21:24.005886+0000","last_became_active":"2024-03-24T11:20:59.940573+0000","last_became_peered":"2024-03-24T11:20:59.940573+0000","last_unstale":"2024-03-24T11:21:24.005886+0000","last_undegraded":"2024-03-24T11:21:24.005886+0000","last_fullsized":"2024-03-24T11:21:24.005886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:31:52.311185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011332+0000","last_change":"2024-03-24T11:20:55.924278+0000","last_active":"2024-03-24T11:21:24.011332+0000","last_peered":"2024-03-24T11:21:24.011332+0000","last_clean":"2024-03-24T11:21:24.011332+0000","last_became_active":"2024-03-24T11:20:55.923862+0000","last_became_peered":"2024-03-24T11:20:55.923862+0000","last_unstale":"2024-03-24T11:21:24.011332+0000","last_undegraded":"2024-03-24T11:21:24.011332+0000","last_fullsized":"2024-03-24T11:21:24.011332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:00:17.503859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010184+0000","last_change":"2024-03-24T11:20:57.935207+0000","last_active":"2024-03-24T11:21:24.010184+0000","last_peered":"2024-03-24T11:21:24.010184+0000","last_clean":"2024-03-24T11:21:24.010184+0000","last_became_active":"2024-03-24T11:20:57.934750+0000","last_became_peered":"2024-03-24T11:20:57.934750+0000","last_unstale":"2024-03-24T11:21:24.010184+0000","last_undegraded":"2024-03-24T11:21:24.010184+0000","last_fullsized":"2024-03-24T11:21:24.010184+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:12.481810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008685+0000","last_change":"2024-03-24T11:21:01.952536+0000","last_active":"2024-03-24T11:21:24.008685+0000","last_peered":"2024-03-24T11:21:24.008685+0000","last_clean":"2024-03-24T11:21:24.008685+0000","last_became_active":"2024-03-24T11:21:01.952148+0000","last_became_peered":"2024-03-24T11:21:01.952148+0000","last_unstale":"2024-03-24T11:21:24.008685+0000","last_undegraded":"2024-03-24T11:21:24.008685+0000","last_fullsized":"2024-03-24T11:21:24.008685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:23:47.016940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006264+0000","last_change":"2024-03-24T11:20:59.941494+0000","last_active":"2024-03-24T11:21:24.006264+0000","last_peered":"2024-03-24T11:21:24.006264+0000","last_clean":"2024-03-24T11:21:24.006264+0000","last_became_active":"2024-03-24T11:20:59.941217+0000","last_became_peered":"2024-03-24T11:20:59.941217+0000","last_unstale":"2024-03-24T11:21:24.006264+0000","last_undegraded":"2024-03-24T11:21:24.006264+0000","last_fullsized":"2024-03-24T11:21:24.006264+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:38:29.466392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009627+0000","last_change":"2024-03-24T11:20:55.925165+0000","last_active":"2024-03-24T11:21:24.009627+0000","last_peered":"2024-03-24T11:21:24.009627+0000","last_clean":"2024-03-24T11:21:24.009627+0000","last_became_active":"2024-03-24T11:20:55.924269+0000","last_became_peered":"2024-03-24T11:20:55.924269+0000","last_unstale":"2024-03-24T11:21:24.009627+0000","last_undegraded":"2024-03-24T11:21:24.009627+0000","last_fullsized":"2024-03-24T11:21:24.009627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:25.947441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008776+0000","last_change":"2024-03-24T11:20:57.933994+0000","last_active":"2024-03-24T11:21:24.008776+0000","last_peered":"2024-03-24T11:21:24.008776+0000","last_clean":"2024-03-24T11:21:24.008776+0000","last_became_active":"2024-03-24T11:20:57.933700+0000","last_became_peered":"2024-03-24T11:20:57.933700+0000","last_unstale":"2024-03-24T11:21:24.008776+0000","last_undegraded":"2024-03-24T11:21:24.008776+0000","last_fullsized":"2024-03-24T11:21:24.008776+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:15:25.373720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007885+0000","last_change":"2024-03-24T11:21:01.955617+0000","last_active":"2024-03-24T11:21:24.007885+0000","last_peered":"2024-03-24T11:21:24.007885+0000","last_clean":"2024-03-24T11:21:24.007885+0000","last_became_active":"2024-03-24T11:21:01.955317+0000","last_became_peered":"2024-03-24T11:21:01.955317+0000","last_unstale":"2024-03-24T11:21:24.007885+0000","last_undegraded":"2024-03-24T11:21:24.007885+0000","last_fullsized":"2024-03-24T11:21:24.007885+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:38.865381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010119+0000","last_change":"2024-03-24T11:20:59.941277+0000","last_active":"2024-03-24T11:21:24.010119+0000","last_peered":"2024-03-24T11:21:24.010119+0000","last_clean":"2024-03-24T11:21:24.010119+0000","last_became_active":"2024-03-24T11:20:59.940683+0000","last_became_peered":"2024-03-24T11:20:59.940683+0000","last_unstale":"2024-03-24T11:21:24.010119+0000","last_undegraded":"2024-03-24T11:21:24.010119+0000","last_fullsized":"2024-03-24T11:21:24.010119+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:32:22.239302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011177+0000","last_change":"2024-03-24T11:20:55.931558+0000","last_active":"2024-03-24T11:21:24.011177+0000","last_peered":"2024-03-24T11:21:24.011177+0000","last_clean":"2024-03-24T11:21:24.011177+0000","last_became_active":"2024-03-24T11:20:55.930826+0000","last_became_peered":"2024-03-24T11:20:55.930826+0000","last_unstale":"2024-03-24T11:21:24.011177+0000","last_undegraded":"2024-03-24T11:21:24.011177+0000","last_fullsized":"2024-03-24T11:21:24.011177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:20:50.998482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009789+0000","last_change":"2024-03-24T11:20:57.935291+0000","last_active":"2024-03-24T11:21:24.009789+0000","last_peered":"2024-03-24T11:21:24.009789+0000","last_clean":"2024-03-24T11:21:24.009789+0000","last_became_active":"2024-03-24T11:20:57.934934+0000","last_became_peered":"2024-03-24T11:20:57.934934+0000","last_unstale":"2024-03-24T11:21:24.009789+0000","last_undegraded":"2024-03-24T11:21:24.009789+0000","last_fullsized":"2024-03-24T11:21:24.009789+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:57:12.936663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009941+0000","last_change":"2024-03-24T11:21:01.951405+0000","last_active":"2024-03-24T11:21:24.009941+0000","last_peered":"2024-03-24T11:21:24.009941+0000","last_clean":"2024-03-24T11:21:24.009941+0000","last_became_active":"2024-03-24T11:21:01.951128+0000","last_became_peered":"2024-03-24T11:21:01.951128+0000","last_unstale":"2024-03-24T11:21:24.009941+0000","last_undegraded":"2024-03-24T11:21:24.009941+0000","last_fullsized":"2024-03-24T11:21:24.009941+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:02:26.401653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006118+0000","last_change":"2024-03-24T11:20:59.945336+0000","last_active":"2024-03-24T11:21:24.006118+0000","last_peered":"2024-03-24T11:21:24.006118+0000","last_clean":"2024-03-24T11:21:24.006118+0000","last_became_active":"2024-03-24T11:20:59.944985+0000","last_became_peered":"2024-03-24T11:20:59.944985+0000","last_unstale":"2024-03-24T11:21:24.006118+0000","last_undegraded":"2024-03-24T11:21:24.006118+0000","last_fullsized":"2024-03-24T11:21:24.006118+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:55.994031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009588+0000","last_change":"2024-03-24T11:20:55.922773+0000","last_active":"2024-03-24T11:21:24.009588+0000","last_peered":"2024-03-24T11:21:24.009588+0000","last_clean":"2024-03-24T11:21:24.009588+0000","last_became_active":"2024-03-24T11:20:55.922391+0000","last_became_peered":"2024-03-24T11:20:55.922391+0000","last_unstale":"2024-03-24T11:21:24.009588+0000","last_undegraded":"2024-03-24T11:21:24.009588+0000","last_fullsized":"2024-03-24T11:21:24.009588+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:41.812024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010982+0000","last_change":"2024-03-24T11:20:57.931713+0000","last_active":"2024-03-24T11:21:24.010982+0000","last_peered":"2024-03-24T11:21:24.010982+0000","last_clean":"2024-03-24T11:21:24.010982+0000","last_became_active":"2024-03-24T11:20:57.931365+0000","last_became_peered":"2024-03-24T11:20:57.931365+0000","last_unstale":"2024-03-24T11:21:24.010982+0000","last_undegraded":"2024-03-24T11:21:24.010982+0000","last_fullsized":"2024-03-24T11:21:24.010982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:02:24.425954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007034+0000","last_change":"2024-03-24T11:21:01.961979+0000","last_active":"2024-03-24T11:21:24.007034+0000","last_peered":"2024-03-24T11:21:24.007034+0000","last_clean":"2024-03-24T11:21:24.007034+0000","last_became_active":"2024-03-24T11:21:01.961559+0000","last_became_peered":"2024-03-24T11:21:01.961559+0000","last_unstale":"2024-03-24T11:21:24.007034+0000","last_undegraded":"2024-03-24T11:21:24.007034+0000","last_fullsized":"2024-03-24T11:21:24.007034+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:54:23.340954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977609+0000","last_change":"2024-03-24T11:20:59.947700+0000","last_active":"2024-03-24T11:21:40.977609+0000","last_peered":"2024-03-24T11:21:40.977609+0000","last_clean":"2024-03-24T11:21:40.977609+0000","last_became_active":"2024-03-24T11:20:59.947339+0000","last_became_peered":"2024-03-24T11:20:59.947339+0000","last_unstale":"2024-03-24T11:21:40.977609+0000","last_undegraded":"2024-03-24T11:21:40.977609+0000","last_fullsized":"2024-03-24T11:21:40.977609+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:03.494195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009004+0000","last_change":"2024-03-24T11:20:55.931064+0000","last_active":"2024-03-24T11:21:24.009004+0000","last_peered":"2024-03-24T11:21:24.009004+0000","last_clean":"2024-03-24T11:21:24.009004+0000","last_became_active":"2024-03-24T11:20:55.930313+0000","last_became_peered":"2024-03-24T11:20:55.930313+0000","last_unstale":"2024-03-24T11:21:24.009004+0000","last_undegraded":"2024-03-24T11:21:24.009004+0000","last_fullsized":"2024-03-24T11:21:24.009004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:02:45.848066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006612+0000","last_change":"2024-03-24T11:20:57.933356+0000","last_active":"2024-03-24T11:21:24.006612+0000","last_peered":"2024-03-24T11:21:24.006612+0000","last_clean":"2024-03-24T11:21:24.006612+0000","last_became_active":"2024-03-24T11:20:57.933060+0000","last_became_peered":"2024-03-24T11:20:57.933060+0000","last_unstale":"2024-03-24T11:21:24.006612+0000","last_undegraded":"2024-03-24T11:21:24.006612+0000","last_fullsized":"2024-03-24T11:21:24.006612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:02:54.414664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008945+0000","last_change":"2024-03-24T11:21:01.957664+0000","last_active":"2024-03-24T11:21:24.008945+0000","last_peered":"2024-03-24T11:21:24.008945+0000","last_clean":"2024-03-24T11:21:24.008945+0000","last_became_active":"2024-03-24T11:21:01.956515+0000","last_became_peered":"2024-03-24T11:21:01.956515+0000","last_unstale":"2024-03-24T11:21:24.008945+0000","last_undegraded":"2024-03-24T11:21:24.008945+0000","last_fullsized":"2024-03-24T11:21:24.008945+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:27:50.546309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977690+0000","last_change":"2024-03-24T11:20:59.945043+0000","last_active":"2024-03-24T11:21:40.977690+0000","last_peered":"2024-03-24T11:21:40.977690+0000","last_clean":"2024-03-24T11:21:40.977690+0000","last_became_active":"2024-03-24T11:20:59.944559+0000","last_became_peered":"2024-03-24T11:20:59.944559+0000","last_unstale":"2024-03-24T11:21:40.977690+0000","last_undegraded":"2024-03-24T11:21:40.977690+0000","last_fullsized":"2024-03-24T11:21:40.977690+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:27:40.580606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008316+0000","last_change":"2024-03-24T11:20:55.930858+0000","last_active":"2024-03-24T11:21:24.008316+0000","last_peered":"2024-03-24T11:21:24.008316+0000","last_clean":"2024-03-24T11:21:24.008316+0000","last_became_active":"2024-03-24T11:20:55.930167+0000","last_became_peered":"2024-03-24T11:20:55.930167+0000","last_unstale":"2024-03-24T11:21:24.008316+0000","last_undegraded":"2024-03-24T11:21:24.008316+0000","last_fullsized":"2024-03-24T11:21:24.008316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:05:43.060323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011289+0000","last_change":"2024-03-24T11:20:57.935977+0000","last_active":"2024-03-24T11:21:24.011289+0000","last_peered":"2024-03-24T11:21:24.011289+0000","last_clean":"2024-03-24T11:21:24.011289+0000","last_became_active":"2024-03-24T11:20:57.935304+0000","last_became_peered":"2024-03-24T11:20:57.935304+0000","last_unstale":"2024-03-24T11:21:24.011289+0000","last_undegraded":"2024-03-24T11:21:24.011289+0000","last_fullsized":"2024-03-24T11:21:24.011289+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:12.750055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009051+0000","last_change":"2024-03-24T11:21:01.957564+0000","last_active":"2024-03-24T11:21:24.009051+0000","last_peered":"2024-03-24T11:21:24.009051+0000","last_clean":"2024-03-24T11:21:24.009051+0000","last_became_active":"2024-03-24T11:21:01.956490+0000","last_became_peered":"2024-03-24T11:21:01.956490+0000","last_unstale":"2024-03-24T11:21:24.009051+0000","last_undegraded":"2024-03-24T11:21:24.009051+0000","last_fullsized":"2024-03-24T11:21:24.009051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:54.577031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008660+0000","last_change":"2024-03-24T11:20:59.940930+0000","last_active":"2024-03-24T11:21:24.008660+0000","last_peered":"2024-03-24T11:21:24.008660+0000","last_clean":"2024-03-24T11:21:24.008660+0000","last_became_active":"2024-03-24T11:20:59.940594+0000","last_became_peered":"2024-03-24T11:20:59.940594+0000","last_unstale":"2024-03-24T11:21:24.008660+0000","last_undegraded":"2024-03-24T11:21:24.008660+0000","last_fullsized":"2024-03-24T11:21:24.008660+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:01.329016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008535+0000","last_change":"2024-03-24T11:20:55.924182+0000","last_active":"2024-03-24T11:21:24.008535+0000","last_peered":"2024-03-24T11:21:24.008535+0000","last_clean":"2024-03-24T11:21:24.008535+0000","last_became_active":"2024-03-24T11:20:55.923796+0000","last_became_peered":"2024-03-24T11:20:55.923796+0000","last_unstale":"2024-03-24T11:21:24.008535+0000","last_undegraded":"2024-03-24T11:21:24.008535+0000","last_fullsized":"2024-03-24T11:21:24.008535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:17.005509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008276+0000","last_change":"2024-03-24T11:20:57.936445+0000","last_active":"2024-03-24T11:21:24.008276+0000","last_peered":"2024-03-24T11:21:24.008276+0000","last_clean":"2024-03-24T11:21:24.008276+0000","last_became_active":"2024-03-24T11:20:57.936005+0000","last_became_peered":"2024-03-24T11:20:57.936005+0000","last_unstale":"2024-03-24T11:21:24.008276+0000","last_undegraded":"2024-03-24T11:21:24.008276+0000","last_fullsized":"2024-03-24T11:21:24.008276+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:02:14.639678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007818+0000","last_change":"2024-03-24T11:21:01.962256+0000","last_active":"2024-03-24T11:21:24.007818+0000","last_peered":"2024-03-24T11:21:24.007818+0000","last_clean":"2024-03-24T11:21:24.007818+0000","last_became_active":"2024-03-24T11:21:01.961982+0000","last_became_peered":"2024-03-24T11:21:01.961982+0000","last_unstale":"2024-03-24T11:21:24.007818+0000","last_undegraded":"2024-03-24T11:21:24.007818+0000","last_fullsized":"2024-03-24T11:21:24.007818+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:38:59.981391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008896+0000","last_change":"2024-03-24T11:20:59.940558+0000","last_active":"2024-03-24T11:21:24.008896+0000","last_peered":"2024-03-24T11:21:24.008896+0000","last_clean":"2024-03-24T11:21:24.008896+0000","last_became_active":"2024-03-24T11:20:59.940067+0000","last_became_peered":"2024-03-24T11:20:59.940067+0000","last_unstale":"2024-03-24T11:21:24.008896+0000","last_undegraded":"2024-03-24T11:21:24.008896+0000","last_fullsized":"2024-03-24T11:21:24.008896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:48.927962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008300+0000","last_change":"2024-03-24T11:20:55.931980+0000","last_active":"2024-03-24T11:21:24.008300+0000","last_peered":"2024-03-24T11:21:24.008300+0000","last_clean":"2024-03-24T11:21:24.008300+0000","last_became_active":"2024-03-24T11:20:55.930931+0000","last_became_peered":"2024-03-24T11:20:55.930931+0000","last_unstale":"2024-03-24T11:21:24.008300+0000","last_undegraded":"2024-03-24T11:21:24.008300+0000","last_fullsized":"2024-03-24T11:21:24.008300+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:44:00.543065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010063+0000","last_change":"2024-03-24T11:20:57.932059+0000","last_active":"2024-03-24T11:21:24.010063+0000","last_peered":"2024-03-24T11:21:24.010063+0000","last_clean":"2024-03-24T11:21:24.010063+0000","last_became_active":"2024-03-24T11:20:57.931593+0000","last_became_peered":"2024-03-24T11:20:57.931593+0000","last_unstale":"2024-03-24T11:21:24.010063+0000","last_undegraded":"2024-03-24T11:21:24.010063+0000","last_fullsized":"2024-03-24T11:21:24.010063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:59:57.854889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008980+0000","last_change":"2024-03-24T11:21:01.955300+0000","last_active":"2024-03-24T11:21:24.008980+0000","last_peered":"2024-03-24T11:21:24.008980+0000","last_clean":"2024-03-24T11:21:24.008980+0000","last_became_active":"2024-03-24T11:21:01.954994+0000","last_became_peered":"2024-03-24T11:21:01.954994+0000","last_unstale":"2024-03-24T11:21:24.008980+0000","last_undegraded":"2024-03-24T11:21:24.008980+0000","last_fullsized":"2024-03-24T11:21:24.008980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:31.123059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009728+0000","last_change":"2024-03-24T11:20:55.931125+0000","last_active":"2024-03-24T11:21:24.009728+0000","last_peered":"2024-03-24T11:21:24.009728+0000","last_clean":"2024-03-24T11:21:24.009728+0000","last_became_active":"2024-03-24T11:20:55.930220+0000","last_became_peered":"2024-03-24T11:20:55.930220+0000","last_unstale":"2024-03-24T11:21:24.009728+0000","last_undegraded":"2024-03-24T11:21:24.009728+0000","last_fullsized":"2024-03-24T11:21:24.009728+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:59.879408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009638+0000","last_change":"2024-03-24T11:20:57.928397+0000","last_active":"2024-03-24T11:21:24.009638+0000","last_peered":"2024-03-24T11:21:24.009638+0000","last_clean":"2024-03-24T11:21:24.009638+0000","last_became_active":"2024-03-24T11:20:57.928084+0000","last_became_peered":"2024-03-24T11:20:57.928084+0000","last_unstale":"2024-03-24T11:21:24.009638+0000","last_undegraded":"2024-03-24T11:21:24.009638+0000","last_fullsized":"2024-03-24T11:21:24.009638+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:24:03.243565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977634+0000","last_change":"2024-03-24T11:20:59.942972+0000","last_active":"2024-03-24T11:21:40.977634+0000","last_peered":"2024-03-24T11:21:40.977634+0000","last_clean":"2024-03-24T11:21:40.977634+0000","last_became_active":"2024-03-24T11:20:59.941315+0000","last_became_peered":"2024-03-24T11:20:59.941315+0000","last_unstale":"2024-03-24T11:21:40.977634+0000","last_undegraded":"2024-03-24T11:21:40.977634+0000","last_fullsized":"2024-03-24T11:21:40.977634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:40:50.599369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006343+0000","last_change":"2024-03-24T11:21:01.959863+0000","last_active":"2024-03-24T11:21:24.006343+0000","last_peered":"2024-03-24T11:21:24.006343+0000","last_clean":"2024-03-24T11:21:24.006343+0000","last_became_active":"2024-03-24T11:21:01.959550+0000","last_became_peered":"2024-03-24T11:21:01.959550+0000","last_unstale":"2024-03-24T11:21:24.006343+0000","last_undegraded":"2024-03-24T11:21:24.006343+0000","last_fullsized":"2024-03-24T11:21:24.006343+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:49.616671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007289+0000","last_change":"2024-03-24T11:20:55.922218+0000","last_active":"2024-03-24T11:21:24.007289+0000","last_peered":"2024-03-24T11:21:24.007289+0000","last_clean":"2024-03-24T11:21:24.007289+0000","last_became_active":"2024-03-24T11:20:55.921917+0000","last_became_peered":"2024-03-24T11:20:55.921917+0000","last_unstale":"2024-03-24T11:21:24.007289+0000","last_undegraded":"2024-03-24T11:21:24.007289+0000","last_fullsized":"2024-03-24T11:21:24.007289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:24:12.604488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:35.976615+0000","last_change":"2024-03-24T11:20:57.935881+0000","last_active":"2024-03-24T11:21:35.976615+0000","last_peered":"2024-03-24T11:21:35.976615+0000","last_clean":"2024-03-24T11:21:35.976615+0000","last_became_active":"2024-03-24T11:20:57.935452+0000","last_became_peered":"2024-03-24T11:20:57.935452+0000","last_unstale":"2024-03-24T11:21:35.976615+0000","last_undegraded":"2024-03-24T11:21:35.976615+0000","last_fullsized":"2024-03-24T11:21:35.976615+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:45.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007960+0000","last_change":"2024-03-24T11:20:59.942622+0000","last_active":"2024-03-24T11:21:24.007960+0000","last_peered":"2024-03-24T11:21:24.007960+0000","last_clean":"2024-03-24T11:21:24.007960+0000","last_became_active":"2024-03-24T11:20:59.941464+0000","last_became_peered":"2024-03-24T11:20:59.941464+0000","last_unstale":"2024-03-24T11:21:24.007960+0000","last_undegraded":"2024-03-24T11:21:24.007960+0000","last_fullsized":"2024-03-24T11:21:24.007960+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:00.233482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006254+0000","last_change":"2024-03-24T11:21:01.961724+0000","last_active":"2024-03-24T11:21:24.006254+0000","last_peered":"2024-03-24T11:21:24.006254+0000","last_clean":"2024-03-24T11:21:24.006254+0000","last_became_active":"2024-03-24T11:21:01.961308+0000","last_became_peered":"2024-03-24T11:21:01.961308+0000","last_unstale":"2024-03-24T11:21:24.006254+0000","last_undegraded":"2024-03-24T11:21:24.006254+0000","last_fullsized":"2024-03-24T11:21:24.006254+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:56:58.580068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009508+0000","last_change":"2024-03-24T11:20:55.922643+0000","last_active":"2024-03-24T11:21:24.009508+0000","last_peered":"2024-03-24T11:21:24.009508+0000","last_clean":"2024-03-24T11:21:24.009508+0000","last_became_active":"2024-03-24T11:20:55.922290+0000","last_became_peered":"2024-03-24T11:20:55.922290+0000","last_unstale":"2024-03-24T11:21:24.009508+0000","last_undegraded":"2024-03-24T11:21:24.009508+0000","last_fullsized":"2024-03-24T11:21:24.009508+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:01.690811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011460+0000","last_change":"2024-03-24T11:20:57.936492+0000","last_active":"2024-03-24T11:21:24.011460+0000","last_peered":"2024-03-24T11:21:24.011460+0000","last_clean":"2024-03-24T11:21:24.011460+0000","last_became_active":"2024-03-24T11:20:57.936093+0000","last_became_peered":"2024-03-24T11:20:57.936093+0000","last_unstale":"2024-03-24T11:21:24.011460+0000","last_undegraded":"2024-03-24T11:21:24.011460+0000","last_fullsized":"2024-03-24T11:21:24.011460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:55:07.516862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007130+0000","last_change":"2024-03-24T11:20:59.945652+0000","last_active":"2024-03-24T11:21:24.007130+0000","last_peered":"2024-03-24T11:21:24.007130+0000","last_clean":"2024-03-24T11:21:24.007130+0000","last_became_active":"2024-03-24T11:20:59.945366+0000","last_became_peered":"2024-03-24T11:20:59.945366+0000","last_unstale":"2024-03-24T11:21:24.007130+0000","last_undegraded":"2024-03-24T11:21:24.007130+0000","last_fullsized":"2024-03-24T11:21:24.007130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:42:49.464529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009335+0000","last_change":"2024-03-24T11:21:01.961587+0000","last_active":"2024-03-24T11:21:24.009335+0000","last_peered":"2024-03-24T11:21:24.009335+0000","last_clean":"2024-03-24T11:21:24.009335+0000","last_became_active":"2024-03-24T11:21:01.960633+0000","last_became_peered":"2024-03-24T11:21:01.960633+0000","last_unstale":"2024-03-24T11:21:24.009335+0000","last_undegraded":"2024-03-24T11:21:24.009335+0000","last_fullsized":"2024-03-24T11:21:24.009335+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:34.656406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007761+0000","last_change":"2024-03-24T11:20:57.932223+0000","last_active":"2024-03-24T11:21:24.007761+0000","last_peered":"2024-03-24T11:21:24.007761+0000","last_clean":"2024-03-24T11:21:24.007761+0000","last_became_active":"2024-03-24T11:20:57.931995+0000","last_became_peered":"2024-03-24T11:20:57.931995+0000","last_unstale":"2024-03-24T11:21:24.007761+0000","last_undegraded":"2024-03-24T11:21:24.007761+0000","last_fullsized":"2024-03-24T11:21:24.007761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:46.333473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006368+0000","last_change":"2024-03-24T11:20:55.923411+0000","last_active":"2024-03-24T11:21:24.006368+0000","last_peered":"2024-03-24T11:21:24.006368+0000","last_clean":"2024-03-24T11:21:24.006368+0000","last_became_active":"2024-03-24T11:20:55.923075+0000","last_became_peered":"2024-03-24T11:20:55.923075+0000","last_unstale":"2024-03-24T11:21:24.006368+0000","last_undegraded":"2024-03-24T11:21:24.006368+0000","last_fullsized":"2024-03-24T11:21:24.006368+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:18:41.536187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008659+0000","last_change":"2024-03-24T11:20:59.940786+0000","last_active":"2024-03-24T11:21:24.008659+0000","last_peered":"2024-03-24T11:21:24.008659+0000","last_clean":"2024-03-24T11:21:24.008659+0000","last_became_active":"2024-03-24T11:20:59.940464+0000","last_became_peered":"2024-03-24T11:20:59.940464+0000","last_unstale":"2024-03-24T11:21:24.008659+0000","last_undegraded":"2024-03-24T11:21:24.008659+0000","last_fullsized":"2024-03-24T11:21:24.008659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:10.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008996+0000","last_change":"2024-03-24T11:21:01.951295+0000","last_active":"2024-03-24T11:21:24.008996+0000","last_peered":"2024-03-24T11:21:24.008996+0000","last_clean":"2024-03-24T11:21:24.008996+0000","last_became_active":"2024-03-24T11:21:01.951034+0000","last_became_peered":"2024-03-24T11:21:01.951034+0000","last_unstale":"2024-03-24T11:21:24.008996+0000","last_undegraded":"2024-03-24T11:21:24.008996+0000","last_fullsized":"2024-03-24T11:21:24.008996+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:21.361568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006296+0000","last_change":"2024-03-24T11:20:57.933304+0000","last_active":"2024-03-24T11:21:24.006296+0000","last_peered":"2024-03-24T11:21:24.006296+0000","last_clean":"2024-03-24T11:21:24.006296+0000","last_became_active":"2024-03-24T11:20:57.933003+0000","last_became_peered":"2024-03-24T11:20:57.933003+0000","last_unstale":"2024-03-24T11:21:24.006296+0000","last_undegraded":"2024-03-24T11:21:24.006296+0000","last_fullsized":"2024-03-24T11:21:24.006296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:13.849923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010994+0000","last_change":"2024-03-24T11:20:55.922827+0000","last_active":"2024-03-24T11:21:24.010994+0000","last_peered":"2024-03-24T11:21:24.010994+0000","last_clean":"2024-03-24T11:21:24.010994+0000","last_became_active":"2024-03-24T11:20:55.922191+0000","last_became_peered":"2024-03-24T11:20:55.922191+0000","last_unstale":"2024-03-24T11:21:24.010994+0000","last_undegraded":"2024-03-24T11:21:24.010994+0000","last_fullsized":"2024-03-24T11:21:24.010994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:44.265108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:26.098306+0000","last_change":"2024-03-24T11:20:44.090995+0000","last_active":"2024-03-24T11:21:26.098306+0000","last_peered":"2024-03-24T11:21:26.098306+0000","last_clean":"2024-03-24T11:21:26.098306+0000","last_became_active":"2024-03-24T11:20:44.090680+0000","last_became_peered":"2024-03-24T11:20:44.090680+0000","last_unstale":"2024-03-24T11:21:26.098306+0000","last_undegraded":"2024-03-24T11:21:26.098306+0000","last_fullsized":"2024-03-24T11:21:26.098306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:43.073909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:43.073909+0000","last_clean_scrub_stamp":"2024-03-24T11:20:43.073909+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:17.681265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010000+0000","last_change":"2024-03-24T11:20:59.945172+0000","last_active":"2024-03-24T11:21:24.010000+0000","last_peered":"2024-03-24T11:21:24.010000+0000","last_clean":"2024-03-24T11:21:24.010000+0000","last_became_active":"2024-03-24T11:20:59.944837+0000","last_became_peered":"2024-03-24T11:20:59.944837+0000","last_unstale":"2024-03-24T11:21:24.010000+0000","last_undegraded":"2024-03-24T11:21:24.010000+0000","last_fullsized":"2024-03-24T11:21:24.010000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:57:10.231178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010824+0000","last_change":"2024-03-24T11:21:01.958130+0000","last_active":"2024-03-24T11:21:24.010824+0000","last_peered":"2024-03-24T11:21:24.010824+0000","last_clean":"2024-03-24T11:21:24.010824+0000","last_became_active":"2024-03-24T11:21:01.956269+0000","last_became_peered":"2024-03-24T11:21:01.956269+0000","last_unstale":"2024-03-24T11:21:24.010824+0000","last_undegraded":"2024-03-24T11:21:24.010824+0000","last_fullsized":"2024-03-24T11:21:24.010824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:24:25.425722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006949+0000","last_change":"2024-03-24T11:20:57.933361+0000","last_active":"2024-03-24T11:21:24.006949+0000","last_peered":"2024-03-24T11:21:24.006949+0000","last_clean":"2024-03-24T11:21:24.006949+0000","last_became_active":"2024-03-24T11:20:57.933068+0000","last_became_peered":"2024-03-24T11:20:57.933068+0000","last_unstale":"2024-03-24T11:21:24.006949+0000","last_undegraded":"2024-03-24T11:21:24.006949+0000","last_fullsized":"2024-03-24T11:21:24.006949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:24.446056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008370+0000","last_change":"2024-03-24T11:20:55.925075+0000","last_active":"2024-03-24T11:21:24.008370+0000","last_peered":"2024-03-24T11:21:24.008370+0000","last_clean":"2024-03-24T11:21:24.008370+0000","last_became_active":"2024-03-24T11:20:55.924034+0000","last_became_peered":"2024-03-24T11:20:55.924034+0000","last_unstale":"2024-03-24T11:21:24.008370+0000","last_undegraded":"2024-03-24T11:21:24.008370+0000","last_fullsized":"2024-03-24T11:21:24.008370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:10:19.366140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008742+0000","last_change":"2024-03-24T11:20:59.946328+0000","last_active":"2024-03-24T11:21:24.008742+0000","last_peered":"2024-03-24T11:21:24.008742+0000","last_clean":"2024-03-24T11:21:24.008742+0000","last_became_active":"2024-03-24T11:20:59.946011+0000","last_became_peered":"2024-03-24T11:20:59.946011+0000","last_unstale":"2024-03-24T11:21:24.008742+0000","last_undegraded":"2024-03-24T11:21:24.008742+0000","last_fullsized":"2024-03-24T11:21:24.008742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:33.775861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009154+0000","last_change":"2024-03-24T11:21:01.952786+0000","last_active":"2024-03-24T11:21:24.009154+0000","last_peered":"2024-03-24T11:21:24.009154+0000","last_clean":"2024-03-24T11:21:24.009154+0000","last_became_active":"2024-03-24T11:21:01.952086+0000","last_became_peered":"2024-03-24T11:21:01.952086+0000","last_unstale":"2024-03-24T11:21:24.009154+0000","last_undegraded":"2024-03-24T11:21:24.009154+0000","last_fullsized":"2024-03-24T11:21:24.009154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:13.746474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011104+0000","last_change":"2024-03-24T11:20:55.931991+0000","last_active":"2024-03-24T11:21:24.011104+0000","last_peered":"2024-03-24T11:21:24.011104+0000","last_clean":"2024-03-24T11:21:24.011104+0000","last_became_active":"2024-03-24T11:20:55.930973+0000","last_became_peered":"2024-03-24T11:20:55.930973+0000","last_unstale":"2024-03-24T11:21:24.011104+0000","last_undegraded":"2024-03-24T11:21:24.011104+0000","last_fullsized":"2024-03-24T11:21:24.011104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:54.068140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977570+0000","last_change":"2024-03-24T11:20:57.933642+0000","last_active":"2024-03-24T11:21:40.977570+0000","last_peered":"2024-03-24T11:21:40.977570+0000","last_clean":"2024-03-24T11:21:40.977570+0000","last_became_active":"2024-03-24T11:20:57.933335+0000","last_became_peered":"2024-03-24T11:20:57.933335+0000","last_unstale":"2024-03-24T11:21:40.977570+0000","last_undegraded":"2024-03-24T11:21:40.977570+0000","last_fullsized":"2024-03-24T11:21:40.977570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:05:27.753569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977614+0000","last_change":"2024-03-24T11:20:59.944883+0000","last_active":"2024-03-24T11:21:40.977614+0000","last_peered":"2024-03-24T11:21:40.977614+0000","last_clean":"2024-03-24T11:21:40.977614+0000","last_became_active":"2024-03-24T11:20:59.944579+0000","last_became_peered":"2024-03-24T11:20:59.944579+0000","last_unstale":"2024-03-24T11:21:40.977614+0000","last_undegraded":"2024-03-24T11:21:40.977614+0000","last_fullsized":"2024-03-24T11:21:40.977614+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:08.249737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008408+0000","last_change":"2024-03-24T11:21:01.952494+0000","last_active":"2024-03-24T11:21:24.008408+0000","last_peered":"2024-03-24T11:21:24.008408+0000","last_clean":"2024-03-24T11:21:24.008408+0000","last_became_active":"2024-03-24T11:21:01.952080+0000","last_became_peered":"2024-03-24T11:21:01.952080+0000","last_unstale":"2024-03-24T11:21:24.008408+0000","last_undegraded":"2024-03-24T11:21:24.008408+0000","last_fullsized":"2024-03-24T11:21:24.008408+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:38.650363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009069+0000","last_change":"2024-03-24T11:20:55.923389+0000","last_active":"2024-03-24T11:21:24.009069+0000","last_peered":"2024-03-24T11:21:24.009069+0000","last_clean":"2024-03-24T11:21:24.009069+0000","last_became_active":"2024-03-24T11:20:55.923074+0000","last_became_peered":"2024-03-24T11:20:55.923074+0000","last_unstale":"2024-03-24T11:21:24.009069+0000","last_undegraded":"2024-03-24T11:21:24.009069+0000","last_fullsized":"2024-03-24T11:21:24.009069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:20:24.037234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009060+0000","last_change":"2024-03-24T11:20:57.934804+0000","last_active":"2024-03-24T11:21:24.009060+0000","last_peered":"2024-03-24T11:21:24.009060+0000","last_clean":"2024-03-24T11:21:24.009060+0000","last_became_active":"2024-03-24T11:20:57.934506+0000","last_became_peered":"2024-03-24T11:20:57.934506+0000","last_unstale":"2024-03-24T11:21:24.009060+0000","last_undegraded":"2024-03-24T11:21:24.009060+0000","last_fullsized":"2024-03-24T11:21:24.009060+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:24.712576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977489+0000","last_change":"2024-03-24T11:20:59.942010+0000","last_active":"2024-03-24T11:21:40.977489+0000","last_peered":"2024-03-24T11:21:40.977489+0000","last_clean":"2024-03-24T11:21:40.977489+0000","last_became_active":"2024-03-24T11:20:59.941734+0000","last_became_peered":"2024-03-24T11:20:59.941734+0000","last_unstale":"2024-03-24T11:21:40.977489+0000","last_undegraded":"2024-03-24T11:21:40.977489+0000","last_fullsized":"2024-03-24T11:21:40.977489+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:03.789838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008857+0000","last_change":"2024-03-24T11:21:01.955157+0000","last_active":"2024-03-24T11:21:24.008857+0000","last_peered":"2024-03-24T11:21:24.008857+0000","last_clean":"2024-03-24T11:21:24.008857+0000","last_became_active":"2024-03-24T11:21:01.954749+0000","last_became_peered":"2024-03-24T11:21:01.954749+0000","last_unstale":"2024-03-24T11:21:24.008857+0000","last_undegraded":"2024-03-24T11:21:24.008857+0000","last_fullsized":"2024-03-24T11:21:24.008857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:58:57.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011085+0000","last_change":"2024-03-24T11:20:55.923935+0000","last_active":"2024-03-24T11:21:24.011085+0000","last_peered":"2024-03-24T11:21:24.011085+0000","last_clean":"2024-03-24T11:21:24.011085+0000","last_became_active":"2024-03-24T11:20:55.922996+0000","last_became_peered":"2024-03-24T11:20:55.922996+0000","last_unstale":"2024-03-24T11:21:24.011085+0000","last_undegraded":"2024-03-24T11:21:24.011085+0000","last_fullsized":"2024-03-24T11:21:24.011085+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:32.202554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009212+0000","last_change":"2024-03-24T11:20:57.935251+0000","last_active":"2024-03-24T11:21:24.009212+0000","last_peered":"2024-03-24T11:21:24.009212+0000","last_clean":"2024-03-24T11:21:24.009212+0000","last_became_active":"2024-03-24T11:20:57.934787+0000","last_became_peered":"2024-03-24T11:20:57.934787+0000","last_unstale":"2024-03-24T11:21:24.009212+0000","last_undegraded":"2024-03-24T11:21:24.009212+0000","last_fullsized":"2024-03-24T11:21:24.009212+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:18.540515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007897+0000","last_change":"2024-03-24T11:20:59.943026+0000","last_active":"2024-03-24T11:21:24.007897+0000","last_peered":"2024-03-24T11:21:24.007897+0000","last_clean":"2024-03-24T11:21:24.007897+0000","last_became_active":"2024-03-24T11:20:59.942667+0000","last_became_peered":"2024-03-24T11:20:59.942667+0000","last_unstale":"2024-03-24T11:21:24.007897+0000","last_undegraded":"2024-03-24T11:21:24.007897+0000","last_fullsized":"2024-03-24T11:21:24.007897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:50.158171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010136+0000","last_change":"2024-03-24T11:21:01.957116+0000","last_active":"2024-03-24T11:21:24.010136+0000","last_peered":"2024-03-24T11:21:24.010136+0000","last_clean":"2024-03-24T11:21:24.010136+0000","last_became_active":"2024-03-24T11:21:01.956106+0000","last_became_peered":"2024-03-24T11:21:01.956106+0000","last_unstale":"2024-03-24T11:21:24.010136+0000","last_undegraded":"2024-03-24T11:21:24.010136+0000","last_fullsized":"2024-03-24T11:21:24.010136+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:27.390077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:35.976648+0000","last_change":"2024-03-24T11:20:59.939477+0000","last_active":"2024-03-24T11:21:35.976648+0000","last_peered":"2024-03-24T11:21:35.976648+0000","last_clean":"2024-03-24T11:21:35.976648+0000","last_became_active":"2024-03-24T11:20:59.939122+0000","last_became_peered":"2024-03-24T11:20:59.939122+0000","last_unstale":"2024-03-24T11:21:35.976648+0000","last_undegraded":"2024-03-24T11:21:35.976648+0000","last_fullsized":"2024-03-24T11:21:35.976648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:00.029926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008492+0000","last_change":"2024-03-24T11:20:55.927325+0000","last_active":"2024-03-24T11:21:24.008492+0000","last_peered":"2024-03-24T11:21:24.008492+0000","last_clean":"2024-03-24T11:21:24.008492+0000","last_became_active":"2024-03-24T11:20:55.926980+0000","last_became_peered":"2024-03-24T11:20:55.926980+0000","last_unstale":"2024-03-24T11:21:24.008492+0000","last_undegraded":"2024-03-24T11:21:24.008492+0000","last_fullsized":"2024-03-24T11:21:24.008492+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:39:44.375087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010205+0000","last_change":"2024-03-24T11:20:57.928356+0000","last_active":"2024-03-24T11:21:24.010205+0000","last_peered":"2024-03-24T11:21:24.010205+0000","last_clean":"2024-03-24T11:21:24.010205+0000","last_became_active":"2024-03-24T11:20:57.928032+0000","last_became_peered":"2024-03-24T11:20:57.928032+0000","last_unstale":"2024-03-24T11:21:24.010205+0000","last_undegraded":"2024-03-24T11:21:24.010205+0000","last_fullsized":"2024-03-24T11:21:24.010205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:01.671364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009527+0000","last_change":"2024-03-24T11:21:01.955449+0000","last_active":"2024-03-24T11:21:24.009527+0000","last_peered":"2024-03-24T11:21:24.009527+0000","last_clean":"2024-03-24T11:21:24.009527+0000","last_became_active":"2024-03-24T11:21:01.954514+0000","last_became_peered":"2024-03-24T11:21:01.954514+0000","last_unstale":"2024-03-24T11:21:24.009527+0000","last_undegraded":"2024-03-24T11:21:24.009527+0000","last_fullsized":"2024-03-24T11:21:24.009527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:47.491811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008447+0000","last_change":"2024-03-24T11:20:59.939212+0000","last_active":"2024-03-24T11:21:24.008447+0000","last_peered":"2024-03-24T11:21:24.008447+0000","last_clean":"2024-03-24T11:21:24.008447+0000","last_became_active":"2024-03-24T11:20:59.938952+0000","last_became_peered":"2024-03-24T11:20:59.938952+0000","last_unstale":"2024-03-24T11:21:24.008447+0000","last_undegraded":"2024-03-24T11:21:24.008447+0000","last_fullsized":"2024-03-24T11:21:24.008447+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:20.037594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006631+0000","last_change":"2024-03-24T11:20:55.925031+0000","last_active":"2024-03-24T11:21:24.006631+0000","last_peered":"2024-03-24T11:21:24.006631+0000","last_clean":"2024-03-24T11:21:24.006631+0000","last_became_active":"2024-03-24T11:20:55.924548+0000","last_became_peered":"2024-03-24T11:20:55.924548+0000","last_unstale":"2024-03-24T11:21:24.006631+0000","last_undegraded":"2024-03-24T11:21:24.006631+0000","last_fullsized":"2024-03-24T11:21:24.006631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:36:48.079453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008391+0000","last_change":"2024-03-24T11:20:57.932103+0000","last_active":"2024-03-24T11:21:24.008391+0000","last_peered":"2024-03-24T11:21:24.008391+0000","last_clean":"2024-03-24T11:21:24.008391+0000","last_became_active":"2024-03-24T11:20:57.931699+0000","last_became_peered":"2024-03-24T11:20:57.931699+0000","last_unstale":"2024-03-24T11:21:24.008391+0000","last_undegraded":"2024-03-24T11:21:24.008391+0000","last_fullsized":"2024-03-24T11:21:24.008391+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:24:39.401087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010012+0000","last_change":"2024-03-24T11:21:01.953302+0000","last_active":"2024-03-24T11:21:24.010012+0000","last_peered":"2024-03-24T11:21:24.010012+0000","last_clean":"2024-03-24T11:21:24.010012+0000","last_became_active":"2024-03-24T11:21:01.952967+0000","last_became_peered":"2024-03-24T11:21:01.952967+0000","last_unstale":"2024-03-24T11:21:24.010012+0000","last_undegraded":"2024-03-24T11:21:24.010012+0000","last_fullsized":"2024-03-24T11:21:24.010012+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:42.752539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008959+0000","last_change":"2024-03-24T11:20:55.930111+0000","last_active":"2024-03-24T11:21:24.008959+0000","last_peered":"2024-03-24T11:21:24.008959+0000","last_clean":"2024-03-24T11:21:24.008959+0000","last_became_active":"2024-03-24T11:20:55.929333+0000","last_became_peered":"2024-03-24T11:20:55.929333+0000","last_unstale":"2024-03-24T11:21:24.008959+0000","last_undegraded":"2024-03-24T11:21:24.008959+0000","last_fullsized":"2024-03-24T11:21:24.008959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:15.792550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006863+0000","last_change":"2024-03-24T11:20:57.932397+0000","last_active":"2024-03-24T11:21:24.006863+0000","last_peered":"2024-03-24T11:21:24.006863+0000","last_clean":"2024-03-24T11:21:24.006863+0000","last_became_active":"2024-03-24T11:20:57.932116+0000","last_became_peered":"2024-03-24T11:20:57.932116+0000","last_unstale":"2024-03-24T11:21:24.006863+0000","last_undegraded":"2024-03-24T11:21:24.006863+0000","last_fullsized":"2024-03-24T11:21:24.006863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:40.419417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009474+0000","last_change":"2024-03-24T11:20:59.945046+0000","last_active":"2024-03-24T11:21:24.009474+0000","last_peered":"2024-03-24T11:21:24.009474+0000","last_clean":"2024-03-24T11:21:24.009474+0000","last_became_active":"2024-03-24T11:20:59.944666+0000","last_became_peered":"2024-03-24T11:20:59.944666+0000","last_unstale":"2024-03-24T11:21:24.009474+0000","last_undegraded":"2024-03-24T11:21:24.009474+0000","last_fullsized":"2024-03-24T11:21:24.009474+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:22.249158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006691+0000","last_change":"2024-03-24T11:21:01.961800+0000","last_active":"2024-03-24T11:21:24.006691+0000","last_peered":"2024-03-24T11:21:24.006691+0000","last_clean":"2024-03-24T11:21:24.006691+0000","last_became_active":"2024-03-24T11:21:01.961358+0000","last_became_peered":"2024-03-24T11:21:01.961358+0000","last_unstale":"2024-03-24T11:21:24.006691+0000","last_undegraded":"2024-03-24T11:21:24.006691+0000","last_fullsized":"2024-03-24T11:21:24.006691+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:07.038114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:35.976604+0000","last_change":"2024-03-24T11:20:59.945397+0000","last_active":"2024-03-24T11:21:35.976604+0000","last_peered":"2024-03-24T11:21:35.976604+0000","last_clean":"2024-03-24T11:21:35.976604+0000","last_became_active":"2024-03-24T11:20:59.945003+0000","last_became_peered":"2024-03-24T11:20:59.945003+0000","last_unstale":"2024-03-24T11:21:35.976604+0000","last_undegraded":"2024-03-24T11:21:35.976604+0000","last_fullsized":"2024-03-24T11:21:35.976604+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:43.260950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010048+0000","last_change":"2024-03-24T11:20:55.930307+0000","last_active":"2024-03-24T11:21:24.010048+0000","last_peered":"2024-03-24T11:21:24.010048+0000","last_clean":"2024-03-24T11:21:24.010048+0000","last_became_active":"2024-03-24T11:20:55.929926+0000","last_became_peered":"2024-03-24T11:20:55.929926+0000","last_unstale":"2024-03-24T11:21:24.010048+0000","last_undegraded":"2024-03-24T11:21:24.010048+0000","last_fullsized":"2024-03-24T11:21:24.010048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:15.743004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009667+0000","last_change":"2024-03-24T11:20:57.936510+0000","last_active":"2024-03-24T11:21:24.009667+0000","last_peered":"2024-03-24T11:21:24.009667+0000","last_clean":"2024-03-24T11:21:24.009667+0000","last_became_active":"2024-03-24T11:20:57.935916+0000","last_became_peered":"2024-03-24T11:20:57.935916+0000","last_unstale":"2024-03-24T11:21:24.009667+0000","last_undegraded":"2024-03-24T11:21:24.009667+0000","last_fullsized":"2024-03-24T11:21:24.009667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:44.197748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008496+0000","last_change":"2024-03-24T11:21:01.953606+0000","last_active":"2024-03-24T11:21:24.008496+0000","last_peered":"2024-03-24T11:21:24.008496+0000","last_clean":"2024-03-24T11:21:24.008496+0000","last_became_active":"2024-03-24T11:21:01.952989+0000","last_became_peered":"2024-03-24T11:21:01.952989+0000","last_unstale":"2024-03-24T11:21:24.008496+0000","last_undegraded":"2024-03-24T11:21:24.008496+0000","last_fullsized":"2024-03-24T11:21:24.008496+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:41:48.488447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010120+0000","last_change":"2024-03-24T11:20:55.930807+0000","last_active":"2024-03-24T11:21:24.010120+0000","last_peered":"2024-03-24T11:21:24.010120+0000","last_clean":"2024-03-24T11:21:24.010120+0000","last_became_active":"2024-03-24T11:20:55.929828+0000","last_became_peered":"2024-03-24T11:20:55.929828+0000","last_unstale":"2024-03-24T11:21:24.010120+0000","last_undegraded":"2024-03-24T11:21:24.010120+0000","last_fullsized":"2024-03-24T11:21:24.010120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:11:50.792212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011119+0000","last_change":"2024-03-24T11:20:57.935333+0000","last_active":"2024-03-24T11:21:24.011119+0000","last_peered":"2024-03-24T11:21:24.011119+0000","last_clean":"2024-03-24T11:21:24.011119+0000","last_became_active":"2024-03-24T11:20:57.934887+0000","last_became_peered":"2024-03-24T11:20:57.934887+0000","last_unstale":"2024-03-24T11:21:24.011119+0000","last_undegraded":"2024-03-24T11:21:24.011119+0000","last_fullsized":"2024-03-24T11:21:24.011119+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:54.072551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009019+0000","last_change":"2024-03-24T11:20:59.941103+0000","last_active":"2024-03-24T11:21:24.009019+0000","last_peered":"2024-03-24T11:21:24.009019+0000","last_clean":"2024-03-24T11:21:24.009019+0000","last_became_active":"2024-03-24T11:20:59.940731+0000","last_became_peered":"2024-03-24T11:20:59.940731+0000","last_unstale":"2024-03-24T11:21:24.009019+0000","last_undegraded":"2024-03-24T11:21:24.009019+0000","last_fullsized":"2024-03-24T11:21:24.009019+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:40:33.102930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005890+0000","last_change":"2024-03-24T11:21:01.960678+0000","last_active":"2024-03-24T11:21:24.005890+0000","last_peered":"2024-03-24T11:21:24.005890+0000","last_clean":"2024-03-24T11:21:24.005890+0000","last_became_active":"2024-03-24T11:21:01.960306+0000","last_became_peered":"2024-03-24T11:21:01.960306+0000","last_unstale":"2024-03-24T11:21:24.005890+0000","last_undegraded":"2024-03-24T11:21:24.005890+0000","last_fullsized":"2024-03-24T11:21:24.005890+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:14.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010970+0000","last_change":"2024-03-24T11:20:55.932306+0000","last_active":"2024-03-24T11:21:24.010970+0000","last_peered":"2024-03-24T11:21:24.010970+0000","last_clean":"2024-03-24T11:21:24.010970+0000","last_became_active":"2024-03-24T11:20:55.931302+0000","last_became_peered":"2024-03-24T11:20:55.931302+0000","last_unstale":"2024-03-24T11:21:24.010970+0000","last_undegraded":"2024-03-24T11:21:24.010970+0000","last_fullsized":"2024-03-24T11:21:24.010970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:07:15.524366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009625+0000","last_change":"2024-03-24T11:20:57.936355+0000","last_active":"2024-03-24T11:21:24.009625+0000","last_peered":"2024-03-24T11:21:24.009625+0000","last_clean":"2024-03-24T11:21:24.009625+0000","last_became_active":"2024-03-24T11:20:57.935833+0000","last_became_peered":"2024-03-24T11:20:57.935833+0000","last_unstale":"2024-03-24T11:21:24.009625+0000","last_undegraded":"2024-03-24T11:21:24.009625+0000","last_fullsized":"2024-03-24T11:21:24.009625+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:40.020956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006509+0000","last_change":"2024-03-24T11:20:59.941488+0000","last_active":"2024-03-24T11:21:24.006509+0000","last_peered":"2024-03-24T11:21:24.006509+0000","last_clean":"2024-03-24T11:21:24.006509+0000","last_became_active":"2024-03-24T11:20:59.941108+0000","last_became_peered":"2024-03-24T11:20:59.941108+0000","last_unstale":"2024-03-24T11:21:24.006509+0000","last_undegraded":"2024-03-24T11:21:24.006509+0000","last_fullsized":"2024-03-24T11:21:24.006509+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:51:11.734611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008290+0000","last_change":"2024-03-24T11:21:01.958457+0000","last_active":"2024-03-24T11:21:24.008290+0000","last_peered":"2024-03-24T11:21:24.008290+0000","last_clean":"2024-03-24T11:21:24.008290+0000","last_became_active":"2024-03-24T11:21:01.957823+0000","last_became_peered":"2024-03-24T11:21:01.957823+0000","last_unstale":"2024-03-24T11:21:24.008290+0000","last_undegraded":"2024-03-24T11:21:24.008290+0000","last_fullsized":"2024-03-24T11:21:24.008290+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:25.359714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009174+0000","last_change":"2024-03-24T11:20:55.927029+0000","last_active":"2024-03-24T11:21:24.009174+0000","last_peered":"2024-03-24T11:21:24.009174+0000","last_clean":"2024-03-24T11:21:24.009174+0000","last_became_active":"2024-03-24T11:20:55.926747+0000","last_became_peered":"2024-03-24T11:20:55.926747+0000","last_unstale":"2024-03-24T11:21:24.009174+0000","last_undegraded":"2024-03-24T11:21:24.009174+0000","last_fullsized":"2024-03-24T11:21:24.009174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:26:50.538561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011129+0000","last_change":"2024-03-24T11:20:57.935388+0000","last_active":"2024-03-24T11:21:24.011129+0000","last_peered":"2024-03-24T11:21:24.011129+0000","last_clean":"2024-03-24T11:21:24.011129+0000","last_became_active":"2024-03-24T11:20:57.934967+0000","last_became_peered":"2024-03-24T11:20:57.934967+0000","last_unstale":"2024-03-24T11:21:24.011129+0000","last_undegraded":"2024-03-24T11:21:24.011129+0000","last_fullsized":"2024-03-24T11:21:24.011129+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:28:28.934407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008659+0000","last_change":"2024-03-24T11:20:59.942993+0000","last_active":"2024-03-24T11:21:24.008659+0000","last_peered":"2024-03-24T11:21:24.008659+0000","last_clean":"2024-03-24T11:21:24.008659+0000","last_became_active":"2024-03-24T11:20:59.942708+0000","last_became_peered":"2024-03-24T11:20:59.942708+0000","last_unstale":"2024-03-24T11:21:24.008659+0000","last_undegraded":"2024-03-24T11:21:24.008659+0000","last_fullsized":"2024-03-24T11:21:24.008659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:13:47.266032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:21:52.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:21:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:52 smithi189 ceph-mon[27694]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:21:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:52 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:52 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:52.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:52 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:52.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:52 smithi184 ceph-mon[25816]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:21:52.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:52 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:52.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:52 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:52.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:52 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:52.952 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:53.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:52 smithi186 ceph-mon[27599]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:21:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:52 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:52 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:52 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:53.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:53.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:53 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:53 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:53 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:54 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:54 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:54 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:54.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:54 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:55.076 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:21:55.078 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2024-03-24T11:21:55.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:54 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:55.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:54 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:55.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:54 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:55.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:54 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:55.291 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:54 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:55.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:54 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:55.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:54 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:21:55.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:54 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:21:55.677 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-24T11:21:54.022470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96612,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126556,"statfs":{"total":287972524032,"available":287873593344,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008153+0000","last_change":"2024-03-24T11:21:01.958959+0000","last_active":"2024-03-24T11:21:24.008153+0000","last_peered":"2024-03-24T11:21:24.008153+0000","last_clean":"2024-03-24T11:21:24.008153+0000","last_became_active":"2024-03-24T11:21:01.958420+0000","last_became_peered":"2024-03-24T11:21:01.958420+0000","last_unstale":"2024-03-24T11:21:24.008153+0000","last_undegraded":"2024-03-24T11:21:24.008153+0000","last_fullsized":"2024-03-24T11:21:24.008153+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:37:14.236555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007790+0000","last_change":"2024-03-24T11:20:59.942774+0000","last_active":"2024-03-24T11:21:24.007790+0000","last_peered":"2024-03-24T11:21:24.007790+0000","last_clean":"2024-03-24T11:21:24.007790+0000","last_became_active":"2024-03-24T11:20:59.941212+0000","last_became_peered":"2024-03-24T11:20:59.941212+0000","last_unstale":"2024-03-24T11:21:24.007790+0000","last_undegraded":"2024-03-24T11:21:24.007790+0000","last_fullsized":"2024-03-24T11:21:24.007790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:14:35.912122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008901+0000","last_change":"2024-03-24T11:20:55.923231+0000","last_active":"2024-03-24T11:21:24.008901+0000","last_peered":"2024-03-24T11:21:24.008901+0000","last_clean":"2024-03-24T11:21:24.008901+0000","last_became_active":"2024-03-24T11:20:55.922900+0000","last_became_peered":"2024-03-24T11:20:55.922900+0000","last_unstale":"2024-03-24T11:21:24.008901+0000","last_undegraded":"2024-03-24T11:21:24.008901+0000","last_fullsized":"2024-03-24T11:21:24.008901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:18.794423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009094+0000","last_change":"2024-03-24T11:20:57.934823+0000","last_active":"2024-03-24T11:21:24.009094+0000","last_peered":"2024-03-24T11:21:24.009094+0000","last_clean":"2024-03-24T11:21:24.009094+0000","last_became_active":"2024-03-24T11:20:57.934393+0000","last_became_peered":"2024-03-24T11:20:57.934393+0000","last_unstale":"2024-03-24T11:21:24.009094+0000","last_undegraded":"2024-03-24T11:21:24.009094+0000","last_fullsized":"2024-03-24T11:21:24.009094+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:23.379426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006465+0000","last_change":"2024-03-24T11:21:01.959790+0000","last_active":"2024-03-24T11:21:24.006465+0000","last_peered":"2024-03-24T11:21:24.006465+0000","last_clean":"2024-03-24T11:21:24.006465+0000","last_became_active":"2024-03-24T11:21:01.959476+0000","last_became_peered":"2024-03-24T11:21:01.959476+0000","last_unstale":"2024-03-24T11:21:24.006465+0000","last_undegraded":"2024-03-24T11:21:24.006465+0000","last_fullsized":"2024-03-24T11:21:24.006465+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:49:03.839785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008586+0000","last_change":"2024-03-24T11:20:59.944547+0000","last_active":"2024-03-24T11:21:24.008586+0000","last_peered":"2024-03-24T11:21:24.008586+0000","last_clean":"2024-03-24T11:21:24.008586+0000","last_became_active":"2024-03-24T11:20:59.943831+0000","last_became_peered":"2024-03-24T11:20:59.943831+0000","last_unstale":"2024-03-24T11:21:24.008586+0000","last_undegraded":"2024-03-24T11:21:24.008586+0000","last_fullsized":"2024-03-24T11:21:24.008586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:32:57.552226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009314+0000","last_change":"2024-03-24T11:20:55.925563+0000","last_active":"2024-03-24T11:21:24.009314+0000","last_peered":"2024-03-24T11:21:24.009314+0000","last_clean":"2024-03-24T11:21:24.009314+0000","last_became_active":"2024-03-24T11:20:55.924799+0000","last_became_peered":"2024-03-24T11:20:55.924799+0000","last_unstale":"2024-03-24T11:21:24.009314+0000","last_undegraded":"2024-03-24T11:21:24.009314+0000","last_fullsized":"2024-03-24T11:21:24.009314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:07.051785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008540+0000","last_change":"2024-03-24T11:20:57.934187+0000","last_active":"2024-03-24T11:21:24.008540+0000","last_peered":"2024-03-24T11:21:24.008540+0000","last_clean":"2024-03-24T11:21:24.008540+0000","last_became_active":"2024-03-24T11:20:57.933902+0000","last_became_peered":"2024-03-24T11:20:57.933902+0000","last_unstale":"2024-03-24T11:21:24.008540+0000","last_undegraded":"2024-03-24T11:21:24.008540+0000","last_fullsized":"2024-03-24T11:21:24.008540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:27.035959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008115+0000","last_change":"2024-03-24T11:21:01.953650+0000","last_active":"2024-03-24T11:21:24.008115+0000","last_peered":"2024-03-24T11:21:24.008115+0000","last_clean":"2024-03-24T11:21:24.008115+0000","last_became_active":"2024-03-24T11:21:01.953361+0000","last_became_peered":"2024-03-24T11:21:01.953361+0000","last_unstale":"2024-03-24T11:21:24.008115+0000","last_undegraded":"2024-03-24T11:21:24.008115+0000","last_fullsized":"2024-03-24T11:21:24.008115+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:57:37.168749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010023+0000","last_change":"2024-03-24T11:20:59.944256+0000","last_active":"2024-03-24T11:21:24.010023+0000","last_peered":"2024-03-24T11:21:24.010023+0000","last_clean":"2024-03-24T11:21:24.010023+0000","last_became_active":"2024-03-24T11:20:59.943832+0000","last_became_peered":"2024-03-24T11:20:59.943832+0000","last_unstale":"2024-03-24T11:21:24.010023+0000","last_undegraded":"2024-03-24T11:21:24.010023+0000","last_fullsized":"2024-03-24T11:21:24.010023+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:36:01.606131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011335+0000","last_change":"2024-03-24T11:20:55.931270+0000","last_active":"2024-03-24T11:21:24.011335+0000","last_peered":"2024-03-24T11:21:24.011335+0000","last_clean":"2024-03-24T11:21:24.011335+0000","last_became_active":"2024-03-24T11:20:55.929673+0000","last_became_peered":"2024-03-24T11:20:55.929673+0000","last_unstale":"2024-03-24T11:21:24.011335+0000","last_undegraded":"2024-03-24T11:21:24.011335+0000","last_fullsized":"2024-03-24T11:21:24.011335+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:22.177943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009791+0000","last_change":"2024-03-24T11:20:57.935522+0000","last_active":"2024-03-24T11:21:24.009791+0000","last_peered":"2024-03-24T11:21:24.009791+0000","last_clean":"2024-03-24T11:21:24.009791+0000","last_became_active":"2024-03-24T11:20:57.935122+0000","last_became_peered":"2024-03-24T11:20:57.935122+0000","last_unstale":"2024-03-24T11:21:24.009791+0000","last_undegraded":"2024-03-24T11:21:24.009791+0000","last_fullsized":"2024-03-24T11:21:24.009791+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:47:18.396876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009156+0000","last_change":"2024-03-24T11:21:01.956657+0000","last_active":"2024-03-24T11:21:24.009156+0000","last_peered":"2024-03-24T11:21:24.009156+0000","last_clean":"2024-03-24T11:21:24.009156+0000","last_became_active":"2024-03-24T11:21:01.956017+0000","last_became_peered":"2024-03-24T11:21:01.956017+0000","last_unstale":"2024-03-24T11:21:24.009156+0000","last_undegraded":"2024-03-24T11:21:24.009156+0000","last_fullsized":"2024-03-24T11:21:24.009156+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:30:53.208222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008835+0000","last_change":"2024-03-24T11:20:59.941189+0000","last_active":"2024-03-24T11:21:24.008835+0000","last_peered":"2024-03-24T11:21:24.008835+0000","last_clean":"2024-03-24T11:21:24.008835+0000","last_became_active":"2024-03-24T11:20:59.940825+0000","last_became_peered":"2024-03-24T11:20:59.940825+0000","last_unstale":"2024-03-24T11:21:24.008835+0000","last_undegraded":"2024-03-24T11:21:24.008835+0000","last_fullsized":"2024-03-24T11:21:24.008835+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:44:10.472366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009737+0000","last_change":"2024-03-24T11:20:55.929779+0000","last_active":"2024-03-24T11:21:24.009737+0000","last_peered":"2024-03-24T11:21:24.009737+0000","last_clean":"2024-03-24T11:21:24.009737+0000","last_became_active":"2024-03-24T11:20:55.929445+0000","last_became_peered":"2024-03-24T11:20:55.929445+0000","last_unstale":"2024-03-24T11:21:24.009737+0000","last_undegraded":"2024-03-24T11:21:24.009737+0000","last_fullsized":"2024-03-24T11:21:24.009737+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:00.920661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011203+0000","last_change":"2024-03-24T11:20:57.931822+0000","last_active":"2024-03-24T11:21:24.011203+0000","last_peered":"2024-03-24T11:21:24.011203+0000","last_clean":"2024-03-24T11:21:24.011203+0000","last_became_active":"2024-03-24T11:20:57.931399+0000","last_became_peered":"2024-03-24T11:20:57.931399+0000","last_unstale":"2024-03-24T11:21:24.011203+0000","last_undegraded":"2024-03-24T11:21:24.011203+0000","last_fullsized":"2024-03-24T11:21:24.011203+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:19:08.715267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009413+0000","last_change":"2024-03-24T11:21:01.957599+0000","last_active":"2024-03-24T11:21:24.009413+0000","last_peered":"2024-03-24T11:21:24.009413+0000","last_clean":"2024-03-24T11:21:24.009413+0000","last_became_active":"2024-03-24T11:21:01.956921+0000","last_became_peered":"2024-03-24T11:21:01.956921+0000","last_unstale":"2024-03-24T11:21:24.009413+0000","last_undegraded":"2024-03-24T11:21:24.009413+0000","last_fullsized":"2024-03-24T11:21:24.009413+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:51:02.097620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010110+0000","last_change":"2024-03-24T11:20:59.947557+0000","last_active":"2024-03-24T11:21:24.010110+0000","last_peered":"2024-03-24T11:21:24.010110+0000","last_clean":"2024-03-24T11:21:24.010110+0000","last_became_active":"2024-03-24T11:20:59.947251+0000","last_became_peered":"2024-03-24T11:20:59.947251+0000","last_unstale":"2024-03-24T11:21:24.010110+0000","last_undegraded":"2024-03-24T11:21:24.010110+0000","last_fullsized":"2024-03-24T11:21:24.010110+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:08:17.836024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010221+0000","last_change":"2024-03-24T11:20:55.930046+0000","last_active":"2024-03-24T11:21:24.010221+0000","last_peered":"2024-03-24T11:21:24.010221+0000","last_clean":"2024-03-24T11:21:24.010221+0000","last_became_active":"2024-03-24T11:20:55.929529+0000","last_became_peered":"2024-03-24T11:20:55.929529+0000","last_unstale":"2024-03-24T11:21:24.010221+0000","last_undegraded":"2024-03-24T11:21:24.010221+0000","last_fullsized":"2024-03-24T11:21:24.010221+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:29:58.266487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009073+0000","last_change":"2024-03-24T11:20:57.928352+0000","last_active":"2024-03-24T11:21:24.009073+0000","last_peered":"2024-03-24T11:21:24.009073+0000","last_clean":"2024-03-24T11:21:24.009073+0000","last_became_active":"2024-03-24T11:20:57.927984+0000","last_became_peered":"2024-03-24T11:20:57.927984+0000","last_unstale":"2024-03-24T11:21:24.009073+0000","last_undegraded":"2024-03-24T11:21:24.009073+0000","last_fullsized":"2024-03-24T11:21:24.009073+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:47.835375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007419+0000","last_change":"2024-03-24T11:21:01.962260+0000","last_active":"2024-03-24T11:21:24.007419+0000","last_peered":"2024-03-24T11:21:24.007419+0000","last_clean":"2024-03-24T11:21:24.007419+0000","last_became_active":"2024-03-24T11:21:01.961596+0000","last_became_peered":"2024-03-24T11:21:01.961596+0000","last_unstale":"2024-03-24T11:21:24.007419+0000","last_undegraded":"2024-03-24T11:21:24.007419+0000","last_fullsized":"2024-03-24T11:21:24.007419+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:41:53.989614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009574+0000","last_change":"2024-03-24T11:20:59.946232+0000","last_active":"2024-03-24T11:21:24.009574+0000","last_peered":"2024-03-24T11:21:24.009574+0000","last_clean":"2024-03-24T11:21:24.009574+0000","last_became_active":"2024-03-24T11:20:59.945921+0000","last_became_peered":"2024-03-24T11:20:59.945921+0000","last_unstale":"2024-03-24T11:21:24.009574+0000","last_undegraded":"2024-03-24T11:21:24.009574+0000","last_fullsized":"2024-03-24T11:21:24.009574+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:33:28.418393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009321+0000","last_change":"2024-03-24T11:20:55.923351+0000","last_active":"2024-03-24T11:21:24.009321+0000","last_peered":"2024-03-24T11:21:24.009321+0000","last_clean":"2024-03-24T11:21:24.009321+0000","last_became_active":"2024-03-24T11:20:55.922749+0000","last_became_peered":"2024-03-24T11:20:55.922749+0000","last_unstale":"2024-03-24T11:21:24.009321+0000","last_undegraded":"2024-03-24T11:21:24.009321+0000","last_fullsized":"2024-03-24T11:21:24.009321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:35:13.612167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009214+0000","last_change":"2024-03-24T11:20:57.931769+0000","last_active":"2024-03-24T11:21:24.009214+0000","last_peered":"2024-03-24T11:21:24.009214+0000","last_clean":"2024-03-24T11:21:24.009214+0000","last_became_active":"2024-03-24T11:20:57.931445+0000","last_became_peered":"2024-03-24T11:20:57.931445+0000","last_unstale":"2024-03-24T11:21:24.009214+0000","last_undegraded":"2024-03-24T11:21:24.009214+0000","last_fullsized":"2024-03-24T11:21:24.009214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:00:57.613138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007780+0000","last_change":"2024-03-24T11:21:01.961129+0000","last_active":"2024-03-24T11:21:24.007780+0000","last_peered":"2024-03-24T11:21:24.007780+0000","last_clean":"2024-03-24T11:21:24.007780+0000","last_became_active":"2024-03-24T11:21:01.960598+0000","last_became_peered":"2024-03-24T11:21:01.960598+0000","last_unstale":"2024-03-24T11:21:24.007780+0000","last_undegraded":"2024-03-24T11:21:24.007780+0000","last_fullsized":"2024-03-24T11:21:24.007780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:28:54.547856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977570+0000","last_change":"2024-03-24T11:20:59.948168+0000","last_active":"2024-03-24T11:21:40.977570+0000","last_peered":"2024-03-24T11:21:40.977570+0000","last_clean":"2024-03-24T11:21:40.977570+0000","last_became_active":"2024-03-24T11:20:59.947331+0000","last_became_peered":"2024-03-24T11:20:59.947331+0000","last_unstale":"2024-03-24T11:21:40.977570+0000","last_undegraded":"2024-03-24T11:21:40.977570+0000","last_fullsized":"2024-03-24T11:21:40.977570+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:16.261126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006794+0000","last_change":"2024-03-24T11:20:55.925062+0000","last_active":"2024-03-24T11:21:24.006794+0000","last_peered":"2024-03-24T11:21:24.006794+0000","last_clean":"2024-03-24T11:21:24.006794+0000","last_became_active":"2024-03-24T11:20:55.924606+0000","last_became_peered":"2024-03-24T11:20:55.924606+0000","last_unstale":"2024-03-24T11:21:24.006794+0000","last_undegraded":"2024-03-24T11:21:24.006794+0000","last_fullsized":"2024-03-24T11:21:24.006794+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:40:56.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005886+0000","last_change":"2024-03-24T11:20:57.934243+0000","last_active":"2024-03-24T11:21:24.005886+0000","last_peered":"2024-03-24T11:21:24.005886+0000","last_clean":"2024-03-24T11:21:24.005886+0000","last_became_active":"2024-03-24T11:20:57.933810+0000","last_became_peered":"2024-03-24T11:20:57.933810+0000","last_unstale":"2024-03-24T11:21:24.005886+0000","last_undegraded":"2024-03-24T11:21:24.005886+0000","last_fullsized":"2024-03-24T11:21:24.005886+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:50:27.488867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009661+0000","last_change":"2024-03-24T11:21:01.958387+0000","last_active":"2024-03-24T11:21:24.009661+0000","last_peered":"2024-03-24T11:21:24.009661+0000","last_clean":"2024-03-24T11:21:24.009661+0000","last_became_active":"2024-03-24T11:21:01.957553+0000","last_became_peered":"2024-03-24T11:21:01.957553+0000","last_unstale":"2024-03-24T11:21:24.009661+0000","last_undegraded":"2024-03-24T11:21:24.009661+0000","last_fullsized":"2024-03-24T11:21:24.009661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.335166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008797+0000","last_change":"2024-03-24T11:20:59.941855+0000","last_active":"2024-03-24T11:21:24.008797+0000","last_peered":"2024-03-24T11:21:24.008797+0000","last_clean":"2024-03-24T11:21:24.008797+0000","last_became_active":"2024-03-24T11:20:59.941069+0000","last_became_peered":"2024-03-24T11:20:59.941069+0000","last_unstale":"2024-03-24T11:21:24.008797+0000","last_undegraded":"2024-03-24T11:21:24.008797+0000","last_fullsized":"2024-03-24T11:21:24.008797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:53.610769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008139+0000","last_change":"2024-03-24T11:20:55.930396+0000","last_active":"2024-03-24T11:21:24.008139+0000","last_peered":"2024-03-24T11:21:24.008139+0000","last_clean":"2024-03-24T11:21:24.008139+0000","last_became_active":"2024-03-24T11:20:55.929648+0000","last_became_peered":"2024-03-24T11:20:55.929648+0000","last_unstale":"2024-03-24T11:21:24.008139+0000","last_undegraded":"2024-03-24T11:21:24.008139+0000","last_fullsized":"2024-03-24T11:21:24.008139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:21.885318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005991+0000","last_change":"2024-03-24T11:20:57.932568+0000","last_active":"2024-03-24T11:21:24.005991+0000","last_peered":"2024-03-24T11:21:24.005991+0000","last_clean":"2024-03-24T11:21:24.005991+0000","last_became_active":"2024-03-24T11:20:57.932231+0000","last_became_peered":"2024-03-24T11:20:57.932231+0000","last_unstale":"2024-03-24T11:21:24.005991+0000","last_undegraded":"2024-03-24T11:21:24.005991+0000","last_fullsized":"2024-03-24T11:21:24.005991+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:07:44.514789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008780+0000","last_change":"2024-03-24T11:21:01.958125+0000","last_active":"2024-03-24T11:21:24.008780+0000","last_peered":"2024-03-24T11:21:24.008780+0000","last_clean":"2024-03-24T11:21:24.008780+0000","last_became_active":"2024-03-24T11:21:01.956305+0000","last_became_peered":"2024-03-24T11:21:01.956305+0000","last_unstale":"2024-03-24T11:21:24.008780+0000","last_undegraded":"2024-03-24T11:21:24.008780+0000","last_fullsized":"2024-03-24T11:21:24.008780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:18:54.892727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005886+0000","last_change":"2024-03-24T11:20:59.941565+0000","last_active":"2024-03-24T11:21:24.005886+0000","last_peered":"2024-03-24T11:21:24.005886+0000","last_clean":"2024-03-24T11:21:24.005886+0000","last_became_active":"2024-03-24T11:20:59.940573+0000","last_became_peered":"2024-03-24T11:20:59.940573+0000","last_unstale":"2024-03-24T11:21:24.005886+0000","last_undegraded":"2024-03-24T11:21:24.005886+0000","last_fullsized":"2024-03-24T11:21:24.005886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:31:52.311185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011332+0000","last_change":"2024-03-24T11:20:55.924278+0000","last_active":"2024-03-24T11:21:24.011332+0000","last_peered":"2024-03-24T11:21:24.011332+0000","last_clean":"2024-03-24T11:21:24.011332+0000","last_became_active":"2024-03-24T11:20:55.923862+0000","last_became_peered":"2024-03-24T11:20:55.923862+0000","last_unstale":"2024-03-24T11:21:24.011332+0000","last_undegraded":"2024-03-24T11:21:24.011332+0000","last_fullsized":"2024-03-24T11:21:24.011332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:00:17.503859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010184+0000","last_change":"2024-03-24T11:20:57.935207+0000","last_active":"2024-03-24T11:21:24.010184+0000","last_peered":"2024-03-24T11:21:24.010184+0000","last_clean":"2024-03-24T11:21:24.010184+0000","last_became_active":"2024-03-24T11:20:57.934750+0000","last_became_peered":"2024-03-24T11:20:57.934750+0000","last_unstale":"2024-03-24T11:21:24.010184+0000","last_undegraded":"2024-03-24T11:21:24.010184+0000","last_fullsized":"2024-03-24T11:21:24.010184+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:12.481810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008685+0000","last_change":"2024-03-24T11:21:01.952536+0000","last_active":"2024-03-24T11:21:24.008685+0000","last_peered":"2024-03-24T11:21:24.008685+0000","last_clean":"2024-03-24T11:21:24.008685+0000","last_became_active":"2024-03-24T11:21:01.952148+0000","last_became_peered":"2024-03-24T11:21:01.952148+0000","last_unstale":"2024-03-24T11:21:24.008685+0000","last_undegraded":"2024-03-24T11:21:24.008685+0000","last_fullsized":"2024-03-24T11:21:24.008685+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:23:47.016940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006264+0000","last_change":"2024-03-24T11:20:59.941494+0000","last_active":"2024-03-24T11:21:24.006264+0000","last_peered":"2024-03-24T11:21:24.006264+0000","last_clean":"2024-03-24T11:21:24.006264+0000","last_became_active":"2024-03-24T11:20:59.941217+0000","last_became_peered":"2024-03-24T11:20:59.941217+0000","last_unstale":"2024-03-24T11:21:24.006264+0000","last_undegraded":"2024-03-24T11:21:24.006264+0000","last_fullsized":"2024-03-24T11:21:24.006264+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:38:29.466392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009627+0000","last_change":"2024-03-24T11:20:55.925165+0000","last_active":"2024-03-24T11:21:24.009627+0000","last_peered":"2024-03-24T11:21:24.009627+0000","last_clean":"2024-03-24T11:21:24.009627+0000","last_became_active":"2024-03-24T11:20:55.924269+0000","last_became_peered":"2024-03-24T11:20:55.924269+0000","last_unstale":"2024-03-24T11:21:24.009627+0000","last_undegraded":"2024-03-24T11:21:24.009627+0000","last_fullsized":"2024-03-24T11:21:24.009627+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:25.947441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008776+0000","last_change":"2024-03-24T11:20:57.933994+0000","last_active":"2024-03-24T11:21:24.008776+0000","last_peered":"2024-03-24T11:21:24.008776+0000","last_clean":"2024-03-24T11:21:24.008776+0000","last_became_active":"2024-03-24T11:20:57.933700+0000","last_became_peered":"2024-03-24T11:20:57.933700+0000","last_unstale":"2024-03-24T11:21:24.008776+0000","last_undegraded":"2024-03-24T11:21:24.008776+0000","last_fullsized":"2024-03-24T11:21:24.008776+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:15:25.373720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007885+0000","last_change":"2024-03-24T11:21:01.955617+0000","last_active":"2024-03-24T11:21:24.007885+0000","last_peered":"2024-03-24T11:21:24.007885+0000","last_clean":"2024-03-24T11:21:24.007885+0000","last_became_active":"2024-03-24T11:21:01.955317+0000","last_became_peered":"2024-03-24T11:21:01.955317+0000","last_unstale":"2024-03-24T11:21:24.007885+0000","last_undegraded":"2024-03-24T11:21:24.007885+0000","last_fullsized":"2024-03-24T11:21:24.007885+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:38.865381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010119+0000","last_change":"2024-03-24T11:20:59.941277+0000","last_active":"2024-03-24T11:21:24.010119+0000","last_peered":"2024-03-24T11:21:24.010119+0000","last_clean":"2024-03-24T11:21:24.010119+0000","last_became_active":"2024-03-24T11:20:59.940683+0000","last_became_peered":"2024-03-24T11:20:59.940683+0000","last_unstale":"2024-03-24T11:21:24.010119+0000","last_undegraded":"2024-03-24T11:21:24.010119+0000","last_fullsized":"2024-03-24T11:21:24.010119+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:32:22.239302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011177+0000","last_change":"2024-03-24T11:20:55.931558+0000","last_active":"2024-03-24T11:21:24.011177+0000","last_peered":"2024-03-24T11:21:24.011177+0000","last_clean":"2024-03-24T11:21:24.011177+0000","last_became_active":"2024-03-24T11:20:55.930826+0000","last_became_peered":"2024-03-24T11:20:55.930826+0000","last_unstale":"2024-03-24T11:21:24.011177+0000","last_undegraded":"2024-03-24T11:21:24.011177+0000","last_fullsized":"2024-03-24T11:21:24.011177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:20:50.998482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009789+0000","last_change":"2024-03-24T11:20:57.935291+0000","last_active":"2024-03-24T11:21:24.009789+0000","last_peered":"2024-03-24T11:21:24.009789+0000","last_clean":"2024-03-24T11:21:24.009789+0000","last_became_active":"2024-03-24T11:20:57.934934+0000","last_became_peered":"2024-03-24T11:20:57.934934+0000","last_unstale":"2024-03-24T11:21:24.009789+0000","last_undegraded":"2024-03-24T11:21:24.009789+0000","last_fullsized":"2024-03-24T11:21:24.009789+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:57:12.936663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009941+0000","last_change":"2024-03-24T11:21:01.951405+0000","last_active":"2024-03-24T11:21:24.009941+0000","last_peered":"2024-03-24T11:21:24.009941+0000","last_clean":"2024-03-24T11:21:24.009941+0000","last_became_active":"2024-03-24T11:21:01.951128+0000","last_became_peered":"2024-03-24T11:21:01.951128+0000","last_unstale":"2024-03-24T11:21:24.009941+0000","last_undegraded":"2024-03-24T11:21:24.009941+0000","last_fullsized":"2024-03-24T11:21:24.009941+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T23:02:26.401653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006118+0000","last_change":"2024-03-24T11:20:59.945336+0000","last_active":"2024-03-24T11:21:24.006118+0000","last_peered":"2024-03-24T11:21:24.006118+0000","last_clean":"2024-03-24T11:21:24.006118+0000","last_became_active":"2024-03-24T11:20:59.944985+0000","last_became_peered":"2024-03-24T11:20:59.944985+0000","last_unstale":"2024-03-24T11:21:24.006118+0000","last_undegraded":"2024-03-24T11:21:24.006118+0000","last_fullsized":"2024-03-24T11:21:24.006118+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:55.994031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009588+0000","last_change":"2024-03-24T11:20:55.922773+0000","last_active":"2024-03-24T11:21:24.009588+0000","last_peered":"2024-03-24T11:21:24.009588+0000","last_clean":"2024-03-24T11:21:24.009588+0000","last_became_active":"2024-03-24T11:20:55.922391+0000","last_became_peered":"2024-03-24T11:20:55.922391+0000","last_unstale":"2024-03-24T11:21:24.009588+0000","last_undegraded":"2024-03-24T11:21:24.009588+0000","last_fullsized":"2024-03-24T11:21:24.009588+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:41.812024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010982+0000","last_change":"2024-03-24T11:20:57.931713+0000","last_active":"2024-03-24T11:21:24.010982+0000","last_peered":"2024-03-24T11:21:24.010982+0000","last_clean":"2024-03-24T11:21:24.010982+0000","last_became_active":"2024-03-24T11:20:57.931365+0000","last_became_peered":"2024-03-24T11:20:57.931365+0000","last_unstale":"2024-03-24T11:21:24.010982+0000","last_undegraded":"2024-03-24T11:21:24.010982+0000","last_fullsized":"2024-03-24T11:21:24.010982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:02:24.425954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007034+0000","last_change":"2024-03-24T11:21:01.961979+0000","last_active":"2024-03-24T11:21:24.007034+0000","last_peered":"2024-03-24T11:21:24.007034+0000","last_clean":"2024-03-24T11:21:24.007034+0000","last_became_active":"2024-03-24T11:21:01.961559+0000","last_became_peered":"2024-03-24T11:21:01.961559+0000","last_unstale":"2024-03-24T11:21:24.007034+0000","last_undegraded":"2024-03-24T11:21:24.007034+0000","last_fullsized":"2024-03-24T11:21:24.007034+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:54:23.340954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977609+0000","last_change":"2024-03-24T11:20:59.947700+0000","last_active":"2024-03-24T11:21:40.977609+0000","last_peered":"2024-03-24T11:21:40.977609+0000","last_clean":"2024-03-24T11:21:40.977609+0000","last_became_active":"2024-03-24T11:20:59.947339+0000","last_became_peered":"2024-03-24T11:20:59.947339+0000","last_unstale":"2024-03-24T11:21:40.977609+0000","last_undegraded":"2024-03-24T11:21:40.977609+0000","last_fullsized":"2024-03-24T11:21:40.977609+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:12:03.494195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009004+0000","last_change":"2024-03-24T11:20:55.931064+0000","last_active":"2024-03-24T11:21:24.009004+0000","last_peered":"2024-03-24T11:21:24.009004+0000","last_clean":"2024-03-24T11:21:24.009004+0000","last_became_active":"2024-03-24T11:20:55.930313+0000","last_became_peered":"2024-03-24T11:20:55.930313+0000","last_unstale":"2024-03-24T11:21:24.009004+0000","last_undegraded":"2024-03-24T11:21:24.009004+0000","last_fullsized":"2024-03-24T11:21:24.009004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:02:45.848066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006612+0000","last_change":"2024-03-24T11:20:57.933356+0000","last_active":"2024-03-24T11:21:24.006612+0000","last_peered":"2024-03-24T11:21:24.006612+0000","last_clean":"2024-03-24T11:21:24.006612+0000","last_became_active":"2024-03-24T11:20:57.933060+0000","last_became_peered":"2024-03-24T11:20:57.933060+0000","last_unstale":"2024-03-24T11:21:24.006612+0000","last_undegraded":"2024-03-24T11:21:24.006612+0000","last_fullsized":"2024-03-24T11:21:24.006612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:02:54.414664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008945+0000","last_change":"2024-03-24T11:21:01.957664+0000","last_active":"2024-03-24T11:21:24.008945+0000","last_peered":"2024-03-24T11:21:24.008945+0000","last_clean":"2024-03-24T11:21:24.008945+0000","last_became_active":"2024-03-24T11:21:01.956515+0000","last_became_peered":"2024-03-24T11:21:01.956515+0000","last_unstale":"2024-03-24T11:21:24.008945+0000","last_undegraded":"2024-03-24T11:21:24.008945+0000","last_fullsized":"2024-03-24T11:21:24.008945+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:27:50.546309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977690+0000","last_change":"2024-03-24T11:20:59.945043+0000","last_active":"2024-03-24T11:21:40.977690+0000","last_peered":"2024-03-24T11:21:40.977690+0000","last_clean":"2024-03-24T11:21:40.977690+0000","last_became_active":"2024-03-24T11:20:59.944559+0000","last_became_peered":"2024-03-24T11:20:59.944559+0000","last_unstale":"2024-03-24T11:21:40.977690+0000","last_undegraded":"2024-03-24T11:21:40.977690+0000","last_fullsized":"2024-03-24T11:21:40.977690+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:27:40.580606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008316+0000","last_change":"2024-03-24T11:20:55.930858+0000","last_active":"2024-03-24T11:21:24.008316+0000","last_peered":"2024-03-24T11:21:24.008316+0000","last_clean":"2024-03-24T11:21:24.008316+0000","last_became_active":"2024-03-24T11:20:55.930167+0000","last_became_peered":"2024-03-24T11:20:55.930167+0000","last_unstale":"2024-03-24T11:21:24.008316+0000","last_undegraded":"2024-03-24T11:21:24.008316+0000","last_fullsized":"2024-03-24T11:21:24.008316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:05:43.060323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011289+0000","last_change":"2024-03-24T11:20:57.935977+0000","last_active":"2024-03-24T11:21:24.011289+0000","last_peered":"2024-03-24T11:21:24.011289+0000","last_clean":"2024-03-24T11:21:24.011289+0000","last_became_active":"2024-03-24T11:20:57.935304+0000","last_became_peered":"2024-03-24T11:20:57.935304+0000","last_unstale":"2024-03-24T11:21:24.011289+0000","last_undegraded":"2024-03-24T11:21:24.011289+0000","last_fullsized":"2024-03-24T11:21:24.011289+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:12.750055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009051+0000","last_change":"2024-03-24T11:21:01.957564+0000","last_active":"2024-03-24T11:21:24.009051+0000","last_peered":"2024-03-24T11:21:24.009051+0000","last_clean":"2024-03-24T11:21:24.009051+0000","last_became_active":"2024-03-24T11:21:01.956490+0000","last_became_peered":"2024-03-24T11:21:01.956490+0000","last_unstale":"2024-03-24T11:21:24.009051+0000","last_undegraded":"2024-03-24T11:21:24.009051+0000","last_fullsized":"2024-03-24T11:21:24.009051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:54.577031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008660+0000","last_change":"2024-03-24T11:20:59.940930+0000","last_active":"2024-03-24T11:21:24.008660+0000","last_peered":"2024-03-24T11:21:24.008660+0000","last_clean":"2024-03-24T11:21:24.008660+0000","last_became_active":"2024-03-24T11:20:59.940594+0000","last_became_peered":"2024-03-24T11:20:59.940594+0000","last_unstale":"2024-03-24T11:21:24.008660+0000","last_undegraded":"2024-03-24T11:21:24.008660+0000","last_fullsized":"2024-03-24T11:21:24.008660+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:01.329016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008535+0000","last_change":"2024-03-24T11:20:55.924182+0000","last_active":"2024-03-24T11:21:24.008535+0000","last_peered":"2024-03-24T11:21:24.008535+0000","last_clean":"2024-03-24T11:21:24.008535+0000","last_became_active":"2024-03-24T11:20:55.923796+0000","last_became_peered":"2024-03-24T11:20:55.923796+0000","last_unstale":"2024-03-24T11:21:24.008535+0000","last_undegraded":"2024-03-24T11:21:24.008535+0000","last_fullsized":"2024-03-24T11:21:24.008535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:17.005509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008276+0000","last_change":"2024-03-24T11:20:57.936445+0000","last_active":"2024-03-24T11:21:24.008276+0000","last_peered":"2024-03-24T11:21:24.008276+0000","last_clean":"2024-03-24T11:21:24.008276+0000","last_became_active":"2024-03-24T11:20:57.936005+0000","last_became_peered":"2024-03-24T11:20:57.936005+0000","last_unstale":"2024-03-24T11:21:24.008276+0000","last_undegraded":"2024-03-24T11:21:24.008276+0000","last_fullsized":"2024-03-24T11:21:24.008276+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:02:14.639678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007818+0000","last_change":"2024-03-24T11:21:01.962256+0000","last_active":"2024-03-24T11:21:24.007818+0000","last_peered":"2024-03-24T11:21:24.007818+0000","last_clean":"2024-03-24T11:21:24.007818+0000","last_became_active":"2024-03-24T11:21:01.961982+0000","last_became_peered":"2024-03-24T11:21:01.961982+0000","last_unstale":"2024-03-24T11:21:24.007818+0000","last_undegraded":"2024-03-24T11:21:24.007818+0000","last_fullsized":"2024-03-24T11:21:24.007818+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:38:59.981391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008896+0000","last_change":"2024-03-24T11:20:59.940558+0000","last_active":"2024-03-24T11:21:24.008896+0000","last_peered":"2024-03-24T11:21:24.008896+0000","last_clean":"2024-03-24T11:21:24.008896+0000","last_became_active":"2024-03-24T11:20:59.940067+0000","last_became_peered":"2024-03-24T11:20:59.940067+0000","last_unstale":"2024-03-24T11:21:24.008896+0000","last_undegraded":"2024-03-24T11:21:24.008896+0000","last_fullsized":"2024-03-24T11:21:24.008896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:48.927962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008300+0000","last_change":"2024-03-24T11:20:55.931980+0000","last_active":"2024-03-24T11:21:24.008300+0000","last_peered":"2024-03-24T11:21:24.008300+0000","last_clean":"2024-03-24T11:21:24.008300+0000","last_became_active":"2024-03-24T11:20:55.930931+0000","last_became_peered":"2024-03-24T11:20:55.930931+0000","last_unstale":"2024-03-24T11:21:24.008300+0000","last_undegraded":"2024-03-24T11:21:24.008300+0000","last_fullsized":"2024-03-24T11:21:24.008300+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:44:00.543065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010063+0000","last_change":"2024-03-24T11:20:57.932059+0000","last_active":"2024-03-24T11:21:24.010063+0000","last_peered":"2024-03-24T11:21:24.010063+0000","last_clean":"2024-03-24T11:21:24.010063+0000","last_became_active":"2024-03-24T11:20:57.931593+0000","last_became_peered":"2024-03-24T11:20:57.931593+0000","last_unstale":"2024-03-24T11:21:24.010063+0000","last_undegraded":"2024-03-24T11:21:24.010063+0000","last_fullsized":"2024-03-24T11:21:24.010063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:59:57.854889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008980+0000","last_change":"2024-03-24T11:21:01.955300+0000","last_active":"2024-03-24T11:21:24.008980+0000","last_peered":"2024-03-24T11:21:24.008980+0000","last_clean":"2024-03-24T11:21:24.008980+0000","last_became_active":"2024-03-24T11:21:01.954994+0000","last_became_peered":"2024-03-24T11:21:01.954994+0000","last_unstale":"2024-03-24T11:21:24.008980+0000","last_undegraded":"2024-03-24T11:21:24.008980+0000","last_fullsized":"2024-03-24T11:21:24.008980+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:31.123059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009728+0000","last_change":"2024-03-24T11:20:55.931125+0000","last_active":"2024-03-24T11:21:24.009728+0000","last_peered":"2024-03-24T11:21:24.009728+0000","last_clean":"2024-03-24T11:21:24.009728+0000","last_became_active":"2024-03-24T11:20:55.930220+0000","last_became_peered":"2024-03-24T11:20:55.930220+0000","last_unstale":"2024-03-24T11:21:24.009728+0000","last_undegraded":"2024-03-24T11:21:24.009728+0000","last_fullsized":"2024-03-24T11:21:24.009728+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:59.879408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009638+0000","last_change":"2024-03-24T11:20:57.928397+0000","last_active":"2024-03-24T11:21:24.009638+0000","last_peered":"2024-03-24T11:21:24.009638+0000","last_clean":"2024-03-24T11:21:24.009638+0000","last_became_active":"2024-03-24T11:20:57.928084+0000","last_became_peered":"2024-03-24T11:20:57.928084+0000","last_unstale":"2024-03-24T11:21:24.009638+0000","last_undegraded":"2024-03-24T11:21:24.009638+0000","last_fullsized":"2024-03-24T11:21:24.009638+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:24:03.243565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977634+0000","last_change":"2024-03-24T11:20:59.942972+0000","last_active":"2024-03-24T11:21:40.977634+0000","last_peered":"2024-03-24T11:21:40.977634+0000","last_clean":"2024-03-24T11:21:40.977634+0000","last_became_active":"2024-03-24T11:20:59.941315+0000","last_became_peered":"2024-03-24T11:20:59.941315+0000","last_unstale":"2024-03-24T11:21:40.977634+0000","last_undegraded":"2024-03-24T11:21:40.977634+0000","last_fullsized":"2024-03-24T11:21:40.977634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:40:50.599369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006343+0000","last_change":"2024-03-24T11:21:01.959863+0000","last_active":"2024-03-24T11:21:24.006343+0000","last_peered":"2024-03-24T11:21:24.006343+0000","last_clean":"2024-03-24T11:21:24.006343+0000","last_became_active":"2024-03-24T11:21:01.959550+0000","last_became_peered":"2024-03-24T11:21:01.959550+0000","last_unstale":"2024-03-24T11:21:24.006343+0000","last_undegraded":"2024-03-24T11:21:24.006343+0000","last_fullsized":"2024-03-24T11:21:24.006343+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:49.616671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007289+0000","last_change":"2024-03-24T11:20:55.922218+0000","last_active":"2024-03-24T11:21:24.007289+0000","last_peered":"2024-03-24T11:21:24.007289+0000","last_clean":"2024-03-24T11:21:24.007289+0000","last_became_active":"2024-03-24T11:20:55.921917+0000","last_became_peered":"2024-03-24T11:20:55.921917+0000","last_unstale":"2024-03-24T11:21:24.007289+0000","last_undegraded":"2024-03-24T11:21:24.007289+0000","last_fullsized":"2024-03-24T11:21:24.007289+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:24:12.604488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977675+0000","last_change":"2024-03-24T11:20:57.935881+0000","last_active":"2024-03-24T11:21:40.977675+0000","last_peered":"2024-03-24T11:21:40.977675+0000","last_clean":"2024-03-24T11:21:40.977675+0000","last_became_active":"2024-03-24T11:20:57.935452+0000","last_became_peered":"2024-03-24T11:20:57.935452+0000","last_unstale":"2024-03-24T11:21:40.977675+0000","last_undegraded":"2024-03-24T11:21:40.977675+0000","last_fullsized":"2024-03-24T11:21:40.977675+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:45.072852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007960+0000","last_change":"2024-03-24T11:20:59.942622+0000","last_active":"2024-03-24T11:21:24.007960+0000","last_peered":"2024-03-24T11:21:24.007960+0000","last_clean":"2024-03-24T11:21:24.007960+0000","last_became_active":"2024-03-24T11:20:59.941464+0000","last_became_peered":"2024-03-24T11:20:59.941464+0000","last_unstale":"2024-03-24T11:21:24.007960+0000","last_undegraded":"2024-03-24T11:21:24.007960+0000","last_fullsized":"2024-03-24T11:21:24.007960+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:00.233482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006254+0000","last_change":"2024-03-24T11:21:01.961724+0000","last_active":"2024-03-24T11:21:24.006254+0000","last_peered":"2024-03-24T11:21:24.006254+0000","last_clean":"2024-03-24T11:21:24.006254+0000","last_became_active":"2024-03-24T11:21:01.961308+0000","last_became_peered":"2024-03-24T11:21:01.961308+0000","last_unstale":"2024-03-24T11:21:24.006254+0000","last_undegraded":"2024-03-24T11:21:24.006254+0000","last_fullsized":"2024-03-24T11:21:24.006254+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:56:58.580068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009508+0000","last_change":"2024-03-24T11:20:55.922643+0000","last_active":"2024-03-24T11:21:24.009508+0000","last_peered":"2024-03-24T11:21:24.009508+0000","last_clean":"2024-03-24T11:21:24.009508+0000","last_became_active":"2024-03-24T11:20:55.922290+0000","last_became_peered":"2024-03-24T11:20:55.922290+0000","last_unstale":"2024-03-24T11:21:24.009508+0000","last_undegraded":"2024-03-24T11:21:24.009508+0000","last_fullsized":"2024-03-24T11:21:24.009508+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:30:01.690811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011460+0000","last_change":"2024-03-24T11:20:57.936492+0000","last_active":"2024-03-24T11:21:24.011460+0000","last_peered":"2024-03-24T11:21:24.011460+0000","last_clean":"2024-03-24T11:21:24.011460+0000","last_became_active":"2024-03-24T11:20:57.936093+0000","last_became_peered":"2024-03-24T11:20:57.936093+0000","last_unstale":"2024-03-24T11:21:24.011460+0000","last_undegraded":"2024-03-24T11:21:24.011460+0000","last_fullsized":"2024-03-24T11:21:24.011460+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:55:07.516862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007130+0000","last_change":"2024-03-24T11:20:59.945652+0000","last_active":"2024-03-24T11:21:24.007130+0000","last_peered":"2024-03-24T11:21:24.007130+0000","last_clean":"2024-03-24T11:21:24.007130+0000","last_became_active":"2024-03-24T11:20:59.945366+0000","last_became_peered":"2024-03-24T11:20:59.945366+0000","last_unstale":"2024-03-24T11:21:24.007130+0000","last_undegraded":"2024-03-24T11:21:24.007130+0000","last_fullsized":"2024-03-24T11:21:24.007130+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:42:49.464529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009335+0000","last_change":"2024-03-24T11:21:01.961587+0000","last_active":"2024-03-24T11:21:24.009335+0000","last_peered":"2024-03-24T11:21:24.009335+0000","last_clean":"2024-03-24T11:21:24.009335+0000","last_became_active":"2024-03-24T11:21:01.960633+0000","last_became_peered":"2024-03-24T11:21:01.960633+0000","last_unstale":"2024-03-24T11:21:24.009335+0000","last_undegraded":"2024-03-24T11:21:24.009335+0000","last_fullsized":"2024-03-24T11:21:24.009335+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:34.656406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007761+0000","last_change":"2024-03-24T11:20:57.932223+0000","last_active":"2024-03-24T11:21:24.007761+0000","last_peered":"2024-03-24T11:21:24.007761+0000","last_clean":"2024-03-24T11:21:24.007761+0000","last_became_active":"2024-03-24T11:20:57.931995+0000","last_became_peered":"2024-03-24T11:20:57.931995+0000","last_unstale":"2024-03-24T11:21:24.007761+0000","last_undegraded":"2024-03-24T11:21:24.007761+0000","last_fullsized":"2024-03-24T11:21:24.007761+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:46.333473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006368+0000","last_change":"2024-03-24T11:20:55.923411+0000","last_active":"2024-03-24T11:21:24.006368+0000","last_peered":"2024-03-24T11:21:24.006368+0000","last_clean":"2024-03-24T11:21:24.006368+0000","last_became_active":"2024-03-24T11:20:55.923075+0000","last_became_peered":"2024-03-24T11:20:55.923075+0000","last_unstale":"2024-03-24T11:21:24.006368+0000","last_undegraded":"2024-03-24T11:21:24.006368+0000","last_fullsized":"2024-03-24T11:21:24.006368+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:18:41.536187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008659+0000","last_change":"2024-03-24T11:20:59.940786+0000","last_active":"2024-03-24T11:21:24.008659+0000","last_peered":"2024-03-24T11:21:24.008659+0000","last_clean":"2024-03-24T11:21:24.008659+0000","last_became_active":"2024-03-24T11:20:59.940464+0000","last_became_peered":"2024-03-24T11:20:59.940464+0000","last_unstale":"2024-03-24T11:21:24.008659+0000","last_undegraded":"2024-03-24T11:21:24.008659+0000","last_fullsized":"2024-03-24T11:21:24.008659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:10.544890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008996+0000","last_change":"2024-03-24T11:21:01.951295+0000","last_active":"2024-03-24T11:21:24.008996+0000","last_peered":"2024-03-24T11:21:24.008996+0000","last_clean":"2024-03-24T11:21:24.008996+0000","last_became_active":"2024-03-24T11:21:01.951034+0000","last_became_peered":"2024-03-24T11:21:01.951034+0000","last_unstale":"2024-03-24T11:21:24.008996+0000","last_undegraded":"2024-03-24T11:21:24.008996+0000","last_fullsized":"2024-03-24T11:21:24.008996+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:55:21.361568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006296+0000","last_change":"2024-03-24T11:20:57.933304+0000","last_active":"2024-03-24T11:21:24.006296+0000","last_peered":"2024-03-24T11:21:24.006296+0000","last_clean":"2024-03-24T11:21:24.006296+0000","last_became_active":"2024-03-24T11:20:57.933003+0000","last_became_peered":"2024-03-24T11:20:57.933003+0000","last_unstale":"2024-03-24T11:21:24.006296+0000","last_undegraded":"2024-03-24T11:21:24.006296+0000","last_fullsized":"2024-03-24T11:21:24.006296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:06:13.849923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010994+0000","last_change":"2024-03-24T11:20:55.922827+0000","last_active":"2024-03-24T11:21:24.010994+0000","last_peered":"2024-03-24T11:21:24.010994+0000","last_clean":"2024-03-24T11:21:24.010994+0000","last_became_active":"2024-03-24T11:20:55.922191+0000","last_became_peered":"2024-03-24T11:20:55.922191+0000","last_unstale":"2024-03-24T11:21:24.010994+0000","last_undegraded":"2024-03-24T11:21:24.010994+0000","last_fullsized":"2024-03-24T11:21:24.010994+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:44.265108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:26.098306+0000","last_change":"2024-03-24T11:20:44.090995+0000","last_active":"2024-03-24T11:21:26.098306+0000","last_peered":"2024-03-24T11:21:26.098306+0000","last_clean":"2024-03-24T11:21:26.098306+0000","last_became_active":"2024-03-24T11:20:44.090680+0000","last_became_peered":"2024-03-24T11:20:44.090680+0000","last_unstale":"2024-03-24T11:21:26.098306+0000","last_undegraded":"2024-03-24T11:21:26.098306+0000","last_fullsized":"2024-03-24T11:21:26.098306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:43.073909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:43.073909+0000","last_clean_scrub_stamp":"2024-03-24T11:20:43.073909+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:17.681265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010000+0000","last_change":"2024-03-24T11:20:59.945172+0000","last_active":"2024-03-24T11:21:24.010000+0000","last_peered":"2024-03-24T11:21:24.010000+0000","last_clean":"2024-03-24T11:21:24.010000+0000","last_became_active":"2024-03-24T11:20:59.944837+0000","last_became_peered":"2024-03-24T11:20:59.944837+0000","last_unstale":"2024-03-24T11:21:24.010000+0000","last_undegraded":"2024-03-24T11:21:24.010000+0000","last_fullsized":"2024-03-24T11:21:24.010000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:57:10.231178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010824+0000","last_change":"2024-03-24T11:21:01.958130+0000","last_active":"2024-03-24T11:21:24.010824+0000","last_peered":"2024-03-24T11:21:24.010824+0000","last_clean":"2024-03-24T11:21:24.010824+0000","last_became_active":"2024-03-24T11:21:01.956269+0000","last_became_peered":"2024-03-24T11:21:01.956269+0000","last_unstale":"2024-03-24T11:21:24.010824+0000","last_undegraded":"2024-03-24T11:21:24.010824+0000","last_fullsized":"2024-03-24T11:21:24.010824+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:24:25.425722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006949+0000","last_change":"2024-03-24T11:20:57.933361+0000","last_active":"2024-03-24T11:21:24.006949+0000","last_peered":"2024-03-24T11:21:24.006949+0000","last_clean":"2024-03-24T11:21:24.006949+0000","last_became_active":"2024-03-24T11:20:57.933068+0000","last_became_peered":"2024-03-24T11:20:57.933068+0000","last_unstale":"2024-03-24T11:21:24.006949+0000","last_undegraded":"2024-03-24T11:21:24.006949+0000","last_fullsized":"2024-03-24T11:21:24.006949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:24.446056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008370+0000","last_change":"2024-03-24T11:20:55.925075+0000","last_active":"2024-03-24T11:21:24.008370+0000","last_peered":"2024-03-24T11:21:24.008370+0000","last_clean":"2024-03-24T11:21:24.008370+0000","last_became_active":"2024-03-24T11:20:55.924034+0000","last_became_peered":"2024-03-24T11:20:55.924034+0000","last_unstale":"2024-03-24T11:21:24.008370+0000","last_undegraded":"2024-03-24T11:21:24.008370+0000","last_fullsized":"2024-03-24T11:21:24.008370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:10:19.366140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008742+0000","last_change":"2024-03-24T11:20:59.946328+0000","last_active":"2024-03-24T11:21:24.008742+0000","last_peered":"2024-03-24T11:21:24.008742+0000","last_clean":"2024-03-24T11:21:24.008742+0000","last_became_active":"2024-03-24T11:20:59.946011+0000","last_became_peered":"2024-03-24T11:20:59.946011+0000","last_unstale":"2024-03-24T11:21:24.008742+0000","last_undegraded":"2024-03-24T11:21:24.008742+0000","last_fullsized":"2024-03-24T11:21:24.008742+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:33.775861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009154+0000","last_change":"2024-03-24T11:21:01.952786+0000","last_active":"2024-03-24T11:21:24.009154+0000","last_peered":"2024-03-24T11:21:24.009154+0000","last_clean":"2024-03-24T11:21:24.009154+0000","last_became_active":"2024-03-24T11:21:01.952086+0000","last_became_peered":"2024-03-24T11:21:01.952086+0000","last_unstale":"2024-03-24T11:21:24.009154+0000","last_undegraded":"2024-03-24T11:21:24.009154+0000","last_fullsized":"2024-03-24T11:21:24.009154+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:13.746474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011104+0000","last_change":"2024-03-24T11:20:55.931991+0000","last_active":"2024-03-24T11:21:24.011104+0000","last_peered":"2024-03-24T11:21:24.011104+0000","last_clean":"2024-03-24T11:21:24.011104+0000","last_became_active":"2024-03-24T11:20:55.930973+0000","last_became_peered":"2024-03-24T11:20:55.930973+0000","last_unstale":"2024-03-24T11:21:24.011104+0000","last_undegraded":"2024-03-24T11:21:24.011104+0000","last_fullsized":"2024-03-24T11:21:24.011104+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:54.068140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977570+0000","last_change":"2024-03-24T11:20:57.933642+0000","last_active":"2024-03-24T11:21:40.977570+0000","last_peered":"2024-03-24T11:21:40.977570+0000","last_clean":"2024-03-24T11:21:40.977570+0000","last_became_active":"2024-03-24T11:20:57.933335+0000","last_became_peered":"2024-03-24T11:20:57.933335+0000","last_unstale":"2024-03-24T11:21:40.977570+0000","last_undegraded":"2024-03-24T11:21:40.977570+0000","last_fullsized":"2024-03-24T11:21:40.977570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:05:27.753569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977614+0000","last_change":"2024-03-24T11:20:59.944883+0000","last_active":"2024-03-24T11:21:40.977614+0000","last_peered":"2024-03-24T11:21:40.977614+0000","last_clean":"2024-03-24T11:21:40.977614+0000","last_became_active":"2024-03-24T11:20:59.944579+0000","last_became_peered":"2024-03-24T11:20:59.944579+0000","last_unstale":"2024-03-24T11:21:40.977614+0000","last_undegraded":"2024-03-24T11:21:40.977614+0000","last_fullsized":"2024-03-24T11:21:40.977614+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:08.249737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008408+0000","last_change":"2024-03-24T11:21:01.952494+0000","last_active":"2024-03-24T11:21:24.008408+0000","last_peered":"2024-03-24T11:21:24.008408+0000","last_clean":"2024-03-24T11:21:24.008408+0000","last_became_active":"2024-03-24T11:21:01.952080+0000","last_became_peered":"2024-03-24T11:21:01.952080+0000","last_unstale":"2024-03-24T11:21:24.008408+0000","last_undegraded":"2024-03-24T11:21:24.008408+0000","last_fullsized":"2024-03-24T11:21:24.008408+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:40:38.650363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009069+0000","last_change":"2024-03-24T11:20:55.923389+0000","last_active":"2024-03-24T11:21:24.009069+0000","last_peered":"2024-03-24T11:21:24.009069+0000","last_clean":"2024-03-24T11:21:24.009069+0000","last_became_active":"2024-03-24T11:20:55.923074+0000","last_became_peered":"2024-03-24T11:20:55.923074+0000","last_unstale":"2024-03-24T11:21:24.009069+0000","last_undegraded":"2024-03-24T11:21:24.009069+0000","last_fullsized":"2024-03-24T11:21:24.009069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:20:24.037234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009060+0000","last_change":"2024-03-24T11:20:57.934804+0000","last_active":"2024-03-24T11:21:24.009060+0000","last_peered":"2024-03-24T11:21:24.009060+0000","last_clean":"2024-03-24T11:21:24.009060+0000","last_became_active":"2024-03-24T11:20:57.934506+0000","last_became_peered":"2024-03-24T11:20:57.934506+0000","last_unstale":"2024-03-24T11:21:24.009060+0000","last_undegraded":"2024-03-24T11:21:24.009060+0000","last_fullsized":"2024-03-24T11:21:24.009060+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:24.712576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977489+0000","last_change":"2024-03-24T11:20:59.942010+0000","last_active":"2024-03-24T11:21:40.977489+0000","last_peered":"2024-03-24T11:21:40.977489+0000","last_clean":"2024-03-24T11:21:40.977489+0000","last_became_active":"2024-03-24T11:20:59.941734+0000","last_became_peered":"2024-03-24T11:20:59.941734+0000","last_unstale":"2024-03-24T11:21:40.977489+0000","last_undegraded":"2024-03-24T11:21:40.977489+0000","last_fullsized":"2024-03-24T11:21:40.977489+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:03.789838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008857+0000","last_change":"2024-03-24T11:21:01.955157+0000","last_active":"2024-03-24T11:21:24.008857+0000","last_peered":"2024-03-24T11:21:24.008857+0000","last_clean":"2024-03-24T11:21:24.008857+0000","last_became_active":"2024-03-24T11:21:01.954749+0000","last_became_peered":"2024-03-24T11:21:01.954749+0000","last_unstale":"2024-03-24T11:21:24.008857+0000","last_undegraded":"2024-03-24T11:21:24.008857+0000","last_fullsized":"2024-03-24T11:21:24.008857+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:58:57.139176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011085+0000","last_change":"2024-03-24T11:20:55.923935+0000","last_active":"2024-03-24T11:21:24.011085+0000","last_peered":"2024-03-24T11:21:24.011085+0000","last_clean":"2024-03-24T11:21:24.011085+0000","last_became_active":"2024-03-24T11:20:55.922996+0000","last_became_peered":"2024-03-24T11:20:55.922996+0000","last_unstale":"2024-03-24T11:21:24.011085+0000","last_undegraded":"2024-03-24T11:21:24.011085+0000","last_fullsized":"2024-03-24T11:21:24.011085+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:32.202554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009212+0000","last_change":"2024-03-24T11:20:57.935251+0000","last_active":"2024-03-24T11:21:24.009212+0000","last_peered":"2024-03-24T11:21:24.009212+0000","last_clean":"2024-03-24T11:21:24.009212+0000","last_became_active":"2024-03-24T11:20:57.934787+0000","last_became_peered":"2024-03-24T11:20:57.934787+0000","last_unstale":"2024-03-24T11:21:24.009212+0000","last_undegraded":"2024-03-24T11:21:24.009212+0000","last_fullsized":"2024-03-24T11:21:24.009212+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:18.540515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.007897+0000","last_change":"2024-03-24T11:20:59.943026+0000","last_active":"2024-03-24T11:21:24.007897+0000","last_peered":"2024-03-24T11:21:24.007897+0000","last_clean":"2024-03-24T11:21:24.007897+0000","last_became_active":"2024-03-24T11:20:59.942667+0000","last_became_peered":"2024-03-24T11:20:59.942667+0000","last_unstale":"2024-03-24T11:21:24.007897+0000","last_undegraded":"2024-03-24T11:21:24.007897+0000","last_fullsized":"2024-03-24T11:21:24.007897+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:50.158171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010136+0000","last_change":"2024-03-24T11:21:01.957116+0000","last_active":"2024-03-24T11:21:24.010136+0000","last_peered":"2024-03-24T11:21:24.010136+0000","last_clean":"2024-03-24T11:21:24.010136+0000","last_became_active":"2024-03-24T11:21:01.956106+0000","last_became_peered":"2024-03-24T11:21:01.956106+0000","last_unstale":"2024-03-24T11:21:24.010136+0000","last_undegraded":"2024-03-24T11:21:24.010136+0000","last_fullsized":"2024-03-24T11:21:24.010136+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:27.390077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977649+0000","last_change":"2024-03-24T11:20:59.939477+0000","last_active":"2024-03-24T11:21:40.977649+0000","last_peered":"2024-03-24T11:21:40.977649+0000","last_clean":"2024-03-24T11:21:40.977649+0000","last_became_active":"2024-03-24T11:20:59.939122+0000","last_became_peered":"2024-03-24T11:20:59.939122+0000","last_unstale":"2024-03-24T11:21:40.977649+0000","last_undegraded":"2024-03-24T11:21:40.977649+0000","last_fullsized":"2024-03-24T11:21:40.977649+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:00.029926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008492+0000","last_change":"2024-03-24T11:20:55.927325+0000","last_active":"2024-03-24T11:21:24.008492+0000","last_peered":"2024-03-24T11:21:24.008492+0000","last_clean":"2024-03-24T11:21:24.008492+0000","last_became_active":"2024-03-24T11:20:55.926980+0000","last_became_peered":"2024-03-24T11:20:55.926980+0000","last_unstale":"2024-03-24T11:21:24.008492+0000","last_undegraded":"2024-03-24T11:21:24.008492+0000","last_fullsized":"2024-03-24T11:21:24.008492+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:39:44.375087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010205+0000","last_change":"2024-03-24T11:20:57.928356+0000","last_active":"2024-03-24T11:21:24.010205+0000","last_peered":"2024-03-24T11:21:24.010205+0000","last_clean":"2024-03-24T11:21:24.010205+0000","last_became_active":"2024-03-24T11:20:57.928032+0000","last_became_peered":"2024-03-24T11:20:57.928032+0000","last_unstale":"2024-03-24T11:21:24.010205+0000","last_undegraded":"2024-03-24T11:21:24.010205+0000","last_fullsized":"2024-03-24T11:21:24.010205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:01.671364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009527+0000","last_change":"2024-03-24T11:21:01.955449+0000","last_active":"2024-03-24T11:21:24.009527+0000","last_peered":"2024-03-24T11:21:24.009527+0000","last_clean":"2024-03-24T11:21:24.009527+0000","last_became_active":"2024-03-24T11:21:01.954514+0000","last_became_peered":"2024-03-24T11:21:01.954514+0000","last_unstale":"2024-03-24T11:21:24.009527+0000","last_undegraded":"2024-03-24T11:21:24.009527+0000","last_fullsized":"2024-03-24T11:21:24.009527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:47.491811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008447+0000","last_change":"2024-03-24T11:20:59.939212+0000","last_active":"2024-03-24T11:21:24.008447+0000","last_peered":"2024-03-24T11:21:24.008447+0000","last_clean":"2024-03-24T11:21:24.008447+0000","last_became_active":"2024-03-24T11:20:59.938952+0000","last_became_peered":"2024-03-24T11:20:59.938952+0000","last_unstale":"2024-03-24T11:21:24.008447+0000","last_undegraded":"2024-03-24T11:21:24.008447+0000","last_fullsized":"2024-03-24T11:21:24.008447+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:12:20.037594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006631+0000","last_change":"2024-03-24T11:20:55.925031+0000","last_active":"2024-03-24T11:21:24.006631+0000","last_peered":"2024-03-24T11:21:24.006631+0000","last_clean":"2024-03-24T11:21:24.006631+0000","last_became_active":"2024-03-24T11:20:55.924548+0000","last_became_peered":"2024-03-24T11:20:55.924548+0000","last_unstale":"2024-03-24T11:21:24.006631+0000","last_undegraded":"2024-03-24T11:21:24.006631+0000","last_fullsized":"2024-03-24T11:21:24.006631+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:36:48.079453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008391+0000","last_change":"2024-03-24T11:20:57.932103+0000","last_active":"2024-03-24T11:21:24.008391+0000","last_peered":"2024-03-24T11:21:24.008391+0000","last_clean":"2024-03-24T11:21:24.008391+0000","last_became_active":"2024-03-24T11:20:57.931699+0000","last_became_peered":"2024-03-24T11:20:57.931699+0000","last_unstale":"2024-03-24T11:21:24.008391+0000","last_undegraded":"2024-03-24T11:21:24.008391+0000","last_fullsized":"2024-03-24T11:21:24.008391+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:24:39.401087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010012+0000","last_change":"2024-03-24T11:21:01.953302+0000","last_active":"2024-03-24T11:21:24.010012+0000","last_peered":"2024-03-24T11:21:24.010012+0000","last_clean":"2024-03-24T11:21:24.010012+0000","last_became_active":"2024-03-24T11:21:01.952967+0000","last_became_peered":"2024-03-24T11:21:01.952967+0000","last_unstale":"2024-03-24T11:21:24.010012+0000","last_undegraded":"2024-03-24T11:21:24.010012+0000","last_fullsized":"2024-03-24T11:21:24.010012+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:42.752539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008959+0000","last_change":"2024-03-24T11:20:55.930111+0000","last_active":"2024-03-24T11:21:24.008959+0000","last_peered":"2024-03-24T11:21:24.008959+0000","last_clean":"2024-03-24T11:21:24.008959+0000","last_became_active":"2024-03-24T11:20:55.929333+0000","last_became_peered":"2024-03-24T11:20:55.929333+0000","last_unstale":"2024-03-24T11:21:24.008959+0000","last_undegraded":"2024-03-24T11:21:24.008959+0000","last_fullsized":"2024-03-24T11:21:24.008959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:15.792550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006863+0000","last_change":"2024-03-24T11:20:57.932397+0000","last_active":"2024-03-24T11:21:24.006863+0000","last_peered":"2024-03-24T11:21:24.006863+0000","last_clean":"2024-03-24T11:21:24.006863+0000","last_became_active":"2024-03-24T11:20:57.932116+0000","last_became_peered":"2024-03-24T11:20:57.932116+0000","last_unstale":"2024-03-24T11:21:24.006863+0000","last_undegraded":"2024-03-24T11:21:24.006863+0000","last_fullsized":"2024-03-24T11:21:24.006863+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:40.419417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009474+0000","last_change":"2024-03-24T11:20:59.945046+0000","last_active":"2024-03-24T11:21:24.009474+0000","last_peered":"2024-03-24T11:21:24.009474+0000","last_clean":"2024-03-24T11:21:24.009474+0000","last_became_active":"2024-03-24T11:20:59.944666+0000","last_became_peered":"2024-03-24T11:20:59.944666+0000","last_unstale":"2024-03-24T11:21:24.009474+0000","last_undegraded":"2024-03-24T11:21:24.009474+0000","last_fullsized":"2024-03-24T11:21:24.009474+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:22.249158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006691+0000","last_change":"2024-03-24T11:21:01.961800+0000","last_active":"2024-03-24T11:21:24.006691+0000","last_peered":"2024-03-24T11:21:24.006691+0000","last_clean":"2024-03-24T11:21:24.006691+0000","last_became_active":"2024-03-24T11:21:01.961358+0000","last_became_peered":"2024-03-24T11:21:01.961358+0000","last_unstale":"2024-03-24T11:21:24.006691+0000","last_undegraded":"2024-03-24T11:21:24.006691+0000","last_fullsized":"2024-03-24T11:21:24.006691+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:07.038114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:40.977659+0000","last_change":"2024-03-24T11:20:59.945397+0000","last_active":"2024-03-24T11:21:40.977659+0000","last_peered":"2024-03-24T11:21:40.977659+0000","last_clean":"2024-03-24T11:21:40.977659+0000","last_became_active":"2024-03-24T11:20:59.945003+0000","last_became_peered":"2024-03-24T11:20:59.945003+0000","last_unstale":"2024-03-24T11:21:40.977659+0000","last_undegraded":"2024-03-24T11:21:40.977659+0000","last_fullsized":"2024-03-24T11:21:40.977659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:43.260950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010048+0000","last_change":"2024-03-24T11:20:55.930307+0000","last_active":"2024-03-24T11:21:24.010048+0000","last_peered":"2024-03-24T11:21:24.010048+0000","last_clean":"2024-03-24T11:21:24.010048+0000","last_became_active":"2024-03-24T11:20:55.929926+0000","last_became_peered":"2024-03-24T11:20:55.929926+0000","last_unstale":"2024-03-24T11:21:24.010048+0000","last_undegraded":"2024-03-24T11:21:24.010048+0000","last_fullsized":"2024-03-24T11:21:24.010048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:15.743004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009667+0000","last_change":"2024-03-24T11:20:57.936510+0000","last_active":"2024-03-24T11:21:24.009667+0000","last_peered":"2024-03-24T11:21:24.009667+0000","last_clean":"2024-03-24T11:21:24.009667+0000","last_became_active":"2024-03-24T11:20:57.935916+0000","last_became_peered":"2024-03-24T11:20:57.935916+0000","last_unstale":"2024-03-24T11:21:24.009667+0000","last_undegraded":"2024-03-24T11:21:24.009667+0000","last_fullsized":"2024-03-24T11:21:24.009667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:44.197748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008496+0000","last_change":"2024-03-24T11:21:01.953606+0000","last_active":"2024-03-24T11:21:24.008496+0000","last_peered":"2024-03-24T11:21:24.008496+0000","last_clean":"2024-03-24T11:21:24.008496+0000","last_became_active":"2024-03-24T11:21:01.952989+0000","last_became_peered":"2024-03-24T11:21:01.952989+0000","last_unstale":"2024-03-24T11:21:24.008496+0000","last_undegraded":"2024-03-24T11:21:24.008496+0000","last_fullsized":"2024-03-24T11:21:24.008496+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:41:48.488447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010120+0000","last_change":"2024-03-24T11:20:55.930807+0000","last_active":"2024-03-24T11:21:24.010120+0000","last_peered":"2024-03-24T11:21:24.010120+0000","last_clean":"2024-03-24T11:21:24.010120+0000","last_became_active":"2024-03-24T11:20:55.929828+0000","last_became_peered":"2024-03-24T11:20:55.929828+0000","last_unstale":"2024-03-24T11:21:24.010120+0000","last_undegraded":"2024-03-24T11:21:24.010120+0000","last_fullsized":"2024-03-24T11:21:24.010120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:11:50.792212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011119+0000","last_change":"2024-03-24T11:20:57.935333+0000","last_active":"2024-03-24T11:21:24.011119+0000","last_peered":"2024-03-24T11:21:24.011119+0000","last_clean":"2024-03-24T11:21:24.011119+0000","last_became_active":"2024-03-24T11:20:57.934887+0000","last_became_peered":"2024-03-24T11:20:57.934887+0000","last_unstale":"2024-03-24T11:21:24.011119+0000","last_undegraded":"2024-03-24T11:21:24.011119+0000","last_fullsized":"2024-03-24T11:21:24.011119+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:54.072551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009019+0000","last_change":"2024-03-24T11:20:59.941103+0000","last_active":"2024-03-24T11:21:24.009019+0000","last_peered":"2024-03-24T11:21:24.009019+0000","last_clean":"2024-03-24T11:21:24.009019+0000","last_became_active":"2024-03-24T11:20:59.940731+0000","last_became_peered":"2024-03-24T11:20:59.940731+0000","last_unstale":"2024-03-24T11:21:24.009019+0000","last_undegraded":"2024-03-24T11:21:24.009019+0000","last_fullsized":"2024-03-24T11:21:24.009019+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:40:33.102930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.005890+0000","last_change":"2024-03-24T11:21:01.960678+0000","last_active":"2024-03-24T11:21:24.005890+0000","last_peered":"2024-03-24T11:21:24.005890+0000","last_clean":"2024-03-24T11:21:24.005890+0000","last_became_active":"2024-03-24T11:21:01.960306+0000","last_became_peered":"2024-03-24T11:21:01.960306+0000","last_unstale":"2024-03-24T11:21:24.005890+0000","last_undegraded":"2024-03-24T11:21:24.005890+0000","last_fullsized":"2024-03-24T11:21:24.005890+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:14.280478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.010970+0000","last_change":"2024-03-24T11:20:55.932306+0000","last_active":"2024-03-24T11:21:24.010970+0000","last_peered":"2024-03-24T11:21:24.010970+0000","last_clean":"2024-03-24T11:21:24.010970+0000","last_became_active":"2024-03-24T11:20:55.931302+0000","last_became_peered":"2024-03-24T11:20:55.931302+0000","last_unstale":"2024-03-24T11:21:24.010970+0000","last_undegraded":"2024-03-24T11:21:24.010970+0000","last_fullsized":"2024-03-24T11:21:24.010970+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:07:15.524366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009625+0000","last_change":"2024-03-24T11:20:57.936355+0000","last_active":"2024-03-24T11:21:24.009625+0000","last_peered":"2024-03-24T11:21:24.009625+0000","last_clean":"2024-03-24T11:21:24.009625+0000","last_became_active":"2024-03-24T11:20:57.935833+0000","last_became_peered":"2024-03-24T11:20:57.935833+0000","last_unstale":"2024-03-24T11:21:24.009625+0000","last_undegraded":"2024-03-24T11:21:24.009625+0000","last_fullsized":"2024-03-24T11:21:24.009625+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:40.020956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.006509+0000","last_change":"2024-03-24T11:20:59.941488+0000","last_active":"2024-03-24T11:21:24.006509+0000","last_peered":"2024-03-24T11:21:24.006509+0000","last_clean":"2024-03-24T11:21:24.006509+0000","last_became_active":"2024-03-24T11:20:59.941108+0000","last_became_peered":"2024-03-24T11:20:59.941108+0000","last_unstale":"2024-03-24T11:21:24.006509+0000","last_undegraded":"2024-03-24T11:21:24.006509+0000","last_fullsized":"2024-03-24T11:21:24.006509+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:51:11.734611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008290+0000","last_change":"2024-03-24T11:21:01.958457+0000","last_active":"2024-03-24T11:21:24.008290+0000","last_peered":"2024-03-24T11:21:24.008290+0000","last_clean":"2024-03-24T11:21:24.008290+0000","last_became_active":"2024-03-24T11:21:01.957823+0000","last_became_peered":"2024-03-24T11:21:01.957823+0000","last_unstale":"2024-03-24T11:21:24.008290+0000","last_undegraded":"2024-03-24T11:21:24.008290+0000","last_fullsized":"2024-03-24T11:21:24.008290+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:21:00.930149+0000","last_clean_scrub_stamp":"2024-03-24T11:21:00.930149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:25.359714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.009174+0000","last_change":"2024-03-24T11:20:55.927029+0000","last_active":"2024-03-24T11:21:24.009174+0000","last_peered":"2024-03-24T11:21:24.009174+0000","last_clean":"2024-03-24T11:21:24.009174+0000","last_became_active":"2024-03-24T11:20:55.926747+0000","last_became_peered":"2024-03-24T11:20:55.926747+0000","last_unstale":"2024-03-24T11:21:24.009174+0000","last_undegraded":"2024-03-24T11:21:24.009174+0000","last_fullsized":"2024-03-24T11:21:24.009174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:54.912638+0000","last_clean_scrub_stamp":"2024-03-24T11:20:54.912638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:26:50.538561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.011129+0000","last_change":"2024-03-24T11:20:57.935388+0000","last_active":"2024-03-24T11:21:24.011129+0000","last_peered":"2024-03-24T11:21:24.011129+0000","last_clean":"2024-03-24T11:21:24.011129+0000","last_became_active":"2024-03-24T11:20:57.934967+0000","last_became_peered":"2024-03-24T11:20:57.934967+0000","last_unstale":"2024-03-24T11:21:24.011129+0000","last_undegraded":"2024-03-24T11:21:24.011129+0000","last_fullsized":"2024-03-24T11:21:24.011129+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:56.915822+0000","last_clean_scrub_stamp":"2024-03-24T11:20:56.915822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:28:28.934407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-24T11:21:24.008659+0000","last_change":"2024-03-24T11:20:59.942993+0000","last_active":"2024-03-24T11:21:24.008659+0000","last_peered":"2024-03-24T11:21:24.008659+0000","last_clean":"2024-03-24T11:21:24.008659+0000","last_became_active":"2024-03-24T11:20:59.942708+0000","last_became_peered":"2024-03-24T11:20:59.942708+0000","last_unstale":"2024-03-24T11:21:24.008659+0000","last_undegraded":"2024-03-24T11:21:24.008659+0000","last_fullsized":"2024-03-24T11:21:24.008659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:20:58.921055+0000","last_clean_scrub_stamp":"2024-03-24T11:20:58.921055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:13:47.266032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:21:55.682 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:21:55.682 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:21:55.683 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:21:55.683 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:21:55.919 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:56.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:55 smithi184 ceph-mon[25816]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:56.033 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:55 smithi184 ceph-mon[25816]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:21:56.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:55 smithi189 ceph-mon[27694]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:56.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:55 smithi189 ceph-mon[27694]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:21:56.283 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:21:56 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:21:56.119Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002285179s 2024-03-24T11:21:56.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:55 smithi186 ceph-mon[27599]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:56.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:55 smithi186 ceph-mon[27599]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:21:56.988 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:56 smithi184 ceph-mon[25816]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:57.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:56 smithi189 ceph-mon[27694]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:57.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:56 smithi186 ceph-mon[27599]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:57.706 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-03-24T11:21:57.706 INFO:teuthology.orchestra.run.smithi184.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:21:57.972 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:57 smithi186 ceph-mon[27599]: from='client.? 172.21.15.184:0/1933012725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:21:58.069 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:57 smithi184 ceph-mon[25816]: from='client.? 172.21.15.184:0/1933012725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:21:58.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:57 smithi189 ceph-mon[27694]: from='client.? 172.21.15.184:0/1933012725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:21:58.387 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:21:58.387 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:21:58.388 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:21:58.398 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi184.front.sepia.ceph.com 2024-03-24T11:21:58.399 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-24T11:21:58.634 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:21:58.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:21:58 smithi186 ceph-mon[27599]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:59.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:21:58 smithi184 ceph-mon[25816]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:21:59.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:21:58 smithi189 ceph-mon[27694]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:00.292 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 systemd[1]: Starting Ceph grafana.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:00.396 INFO:teuthology.orchestra.run.smithi184.stdout:167 167 2024-03-24T11:22:00.668 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 podman[38482]: 2024-03-24 11:22:00.238220606 +0000 UTC m=+0.018055521 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-24T11:22:00.668 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 podman[38482]: 2024-03-24 11:22:00.341508517 +0000 UTC m=+0.121343476 container create f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a, release=239.1651231664, io.openshift.expose-services=, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-24T11:22:00.668 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 podman[38482]: 2024-03-24 11:22:00.560141683 +0000 UTC m=+0.339976608 container init f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, name=ubi8, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, version=8.5, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, distribution-scope=public) 2024-03-24T11:22:00.668 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 podman[38482]: 2024-03-24 11:22:00.565227269 +0000 UTC m=+0.345062184 container start f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a, release=239.1651231664, version=8.5, architecture=x86_64, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, distribution-scope=public, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner ) 2024-03-24T11:22:00.919 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 bash[38482]: f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 2024-03-24T11:22:00.919 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 systemd[1]: Started Ceph grafana.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:00.919 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.724963244Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T11:22:00.919 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725160189Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725178322Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725185351Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725191299Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725197116Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725202588Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725208257Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725214557Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725220755Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725227168Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725232679Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725238399Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=settings t=2024-03-24T11:22:00.725244097Z level=info msg="App mode production" 2024-03-24T11:22:00.920 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=sqlstore t=2024-03-24T11:22:00.725296993Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T11:22:00.921 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=sqlstore t=2024-03-24T11:22:00.725314706Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T11:22:00.921 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.726239338Z level=info msg="Starting DB migrations" 2024-03-24T11:22:00.921 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.726318398Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.726827229Z level=info msg="Executing migration" id="create user table" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.727613448Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.730033562Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.732733164Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.735431106Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.736268803Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.737147118Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.737963014Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T11:22:00.922 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.738361807Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.739266288Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.740072102Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.740931224Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.741751945Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.742541262Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.742962626Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.743840081Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.744637858Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.745507528Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.746341781Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.747491688Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.748387185Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.749203908Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T11:22:00.923 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.750045854Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.750895034Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.751772511Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.752611345Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.752994026Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.753834525Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.754715859Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.755562253Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.756225964Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.757055021Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.757913784Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.758740753Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.759168897Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.76007152Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.760898937Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.76174714Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T11:22:00.924 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.762627467Z level=info msg="Executing migration" id="create star table" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.763492242Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.766326284Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.769255536Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.772158211Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.774860832Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.777682905Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.78041983Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.783258347Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.786021878Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.788829225Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.791683734Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.794653823Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.797559211Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T11:22:00.925 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.800254651Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.803041006Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.805869121Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.80880679Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.811578649Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.814338632Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.817100961Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.819729397Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.822393797Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.825125607Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.827803655Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.830568293Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.833427253Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.836211544Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T11:22:00.926 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.838849771Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.841738049Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.844354601Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.846869403Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.849487674Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.852313336Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.855058283Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.855729695Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.858480086Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.861177756Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.863848786Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.866501758Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.86919958Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T11:22:00.927 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.870125634Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.870940111Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.871804258Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.873158707Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.874045816Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.875050786Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.875797903Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.876663255Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.877539664Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.87830139Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.879268043Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.880099799Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.880855748Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.881753059Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T11:22:00.928 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.882626616Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.883450616Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.88423555Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.88469962Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.885546099Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.886392462Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.887513064Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.888390135Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.889213488Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.890078903Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.890949897Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.89188795Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.892622869Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.893516903Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T11:22:00.929 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.894028008Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.894593279Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.89552975Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.896143875Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.896723035Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.89755174Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.898548586Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.899339289Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.90208808Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.904860129Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.907764797Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.910551254Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.913375396Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.916163523Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T11:22:00.930 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.91914316Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T11:22:00.931 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.92038018Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T11:22:01.067 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:00 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:22:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:01.096 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.923207251Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.926188804Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.929130205Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.929588438Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.929878791Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.932404011Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T11:22:01.172 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.935308692Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.938145814Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.938714719Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.941611285Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.942403192Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.944274956Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.946973667Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.94742385Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.949879866Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.95034014Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.952751615Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T11:22:01.173 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.955196735Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.955558556Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.958382167Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.961205883Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.963724395Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.964084396Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.966879568Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.969466529Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.969793437Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.972326432Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T11:22:01.174 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.975046892Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.977730557Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.978060814Z level=info msg="Executing migration" id="create session table" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.978545322Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.978840828Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.979298776Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.979711186Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.980512933Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.980850761Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.981198195Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T11:22:01.175 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.981521747Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T11:22:01.177 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.981834869Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T11:22:01.177 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.982290205Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T11:22:01.177 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.982575897Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T11:22:01.177 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.983207928Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.983535573Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.984152396Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.985104097Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.985644374Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.986175117Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.986977783Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.987921303Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.988645064Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.989087338Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.989514056Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.989899611Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T11:22:01.178 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.99034555Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.990735566Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.992138476Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.992544402Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.992988182Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.993324459Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.993671025Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.994089501Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T11:22:01.179 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.994744266Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.995425762Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.996110947Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.996798626Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.997256318Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.997561053Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.997844146Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.998263987Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.998704385Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.999113496Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.99952089Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:00 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:00.999931648Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.000580749Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T11:22:01.180 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.001233946Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.001581352Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.002023948Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.002439816Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.003089166Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.003424354Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.003853444Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.00429096Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T11:22:01.181 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.004589224Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.005035928Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.005467521Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.005895875Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.006351511Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.006825314Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.007320017Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.008119448Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.008797655Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.009778408Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.010585103Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.01163738Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T11:22:01.182 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.012466406Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.01317301Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.014743568Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.015586174Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.016466893Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.017310393Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.018129526Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.018982695Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.019814022Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.020685485Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T11:22:01.183 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.021148665Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.022139882Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.022851478Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.02370298Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.02417187Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.025010874Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.025880461Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.026758426Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.027207497Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.028173272Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.028925378Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.029872727Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T11:22:01.184 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.030418648Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.031263843Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.031718655Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.032156876Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.032991025Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.033432613Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.034248993Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.034596423Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.035100336Z level=info msg="Executing migration" id="create team table" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.035504333Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.03600668Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.036500154Z level=info msg="Executing migration" id="create team member table" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.03933193Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.042116944Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T11:22:01.185 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.044954917Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.045388314Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.048304272Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.051119842Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.053862636Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.056549252Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.059288122Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.062050191Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.064973493Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.067796737Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.070704255Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.07341307Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.073897786Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T11:22:01.186 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.07668097Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.079394861Z level=info msg="Executing migration" id="create tag table" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.082156717Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.082774064Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.085522818Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.088219719Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.090855383Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.093966218Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.096811564Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.099382798Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.102050247Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.102439273Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.105253941Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.105878342Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T11:22:01.187 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.10801226Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.110835311Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.111687278Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.114882284Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.117034315Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.11797022Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.118744276Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.119578906Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.120402069Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.12120937Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.121634699Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.122535796Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.123305754Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.12413338Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.124545344Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T11:22:01.188 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.125536845Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.126359687Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.127046623Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.127346183Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.128258979Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.12890856Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.129347632Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.130178817Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.130565763Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.131422736Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.132891157Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.133361014Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.1341747Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.134952992Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T11:22:01.189 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.135815755Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.136765674Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.137510107Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.137978779Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.138935192Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.139680041Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.140542166Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.140975691Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.14181446Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.142700132Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.143544581Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.143997041Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.146915233Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.150525132Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T11:22:01.190 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.153963714Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.154430665Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.158124504Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.158957974Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.160241445Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.162952474Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.165997759Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.168960496Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.171615672Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.174355656Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T11:22:01.191 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.175220545Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T11:22:01.361 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:01.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.178017458Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T11:22:01.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.181011494Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T11:22:01.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.183634332Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T11:22:01.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.184462316Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T11:22:01.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.18725354Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T11:22:01.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.189847299Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.192612971Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.193589526Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.196355005Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.199169142Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.201856663Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.202547878Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.205289071Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.206727798Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.208049367Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.210820825Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.211724934Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.212580163Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T11:22:01.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.213400541Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.213813416Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.214671319Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.215482477Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.21637057Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.216832596Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.217674975Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.218101969Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.218548455Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.219411518Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.219812101Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.220284489Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.220681418Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.221131688Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T11:22:01.448 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.221443564Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T11:22:01.449 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.221856446Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T11:22:01.449 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.222458877Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.222567637Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.222888735Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.223357108Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.223756945Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.223874298Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.22403113Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.234294133Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.234763524Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T11:22:01.450 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.235567911Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.235993811Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.236439664Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.236900827Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.237709178Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.238105549Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.23853071Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.238998211Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.247018415Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.248492366Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.248930782Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.256878305Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.26470226Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T11:22:01.451 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.265284139Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.26584316Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.266312728Z level=info msg="Executing migration" id="create permission table" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.266845382Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.267454454Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.268056133Z level=info msg="Executing migration" id="create role table" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.268598761Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.27006582Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.271331685Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.271752448Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.272198196Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.272611304Z level=info msg="Executing migration" id="create team role table" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.272986992Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.273401903Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.273893482Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T11:22:01.452 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.27432194Z level=info msg="Executing migration" id="create user role table" 2024-03-24T11:22:01.453 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.274698013Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T11:22:01.453 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.275119289Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T11:22:01.453 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.275549519Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T11:22:01.453 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.275980362Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T11:22:01.453 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.27641281Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T11:22:01.453 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.276876607Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.277328675Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.278312569Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.278803352Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.279279814Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.280115909Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.280954562Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.281808592Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.282271008Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.283280687Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.284147496Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.28498786Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T11:22:01.454 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.285286248Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.286136729Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.286628552Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.286982855Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.287297663Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.287864935Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.28828359Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.288728413Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.289690807Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.290526652Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.291417632Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.291900139Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.294635238Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.297470386Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T11:22:01.455 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.300267477Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.300739468Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.300830347Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.303585534Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.303692204Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.30400076Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.304414698Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.304816525Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.305289251Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.307686184Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.310608695Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.313531772Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.3139552Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.316743921Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T11:22:01.456 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.319616378Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.322459971Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.322922721Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.325947616Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.328647905Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.331322432Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.331658261Z level=info msg="Executing migration" id="add share column" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.33267312Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.335452146Z level=info msg="Executing migration" id="create file table" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.335915725Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.336371569Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.339049357Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.34172469Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T11:22:01.457 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.342204422Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.34250612Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.342924727Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.34333565Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.343889064Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.34516915Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.345480559Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.345923548Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.346301508Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.346581836Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.346932495Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.347935925Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.350626644Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.351095954Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T11:22:01.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.360868404Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.363670903Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.364134801Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.367604699Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.370200354Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.370580185Z level=info msg="Executing migration" id="create folder table" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.370960169Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.371448599Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.37189228Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.372199033Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=migrator t=2024-03-24T11:22:01.372639709Z level=info msg="migrations completed" performed=485 skipped=0 duration=646.344286ms 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=sqlstore t=2024-03-24T11:22:01.373030529Z level=info msg="Created default organization" 2024-03-24T11:22:01.459 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=plugin.loader t=2024-03-24T11:22:01.446066241Z level=info msg="Plugin registered" pluginID=input 2024-03-24T11:22:01.727 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=plugin.loader t=2024-03-24T11:22:01.483363137Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T11:22:01.727 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=plugin.loader t=2024-03-24T11:22:01.483394967Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T11:22:01.727 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=secrets t=2024-03-24T11:22:01.483434732Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T11:22:01.727 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=query_data t=2024-03-24T11:22:01.48563866Z level=info msg="Query Service initialization" 2024-03-24T11:22:01.727 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=live.push_http t=2024-03-24T11:22:01.488088583Z level=info msg="Live Push Gateway initialization" 2024-03-24T11:22:01.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:01 smithi184 ceph-mon[25816]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:01.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:01 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:01.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:01 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:01.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:01 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:01.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:01 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:01.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:01 smithi184 ceph-mon[25816]: Deploying daemon node-exporter.a on smithi184 2024-03-24T11:22:02.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-mon[27599]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:02.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:01 smithi186 ceph-mon[27599]: Deploying daemon node-exporter.a on smithi184 2024-03-24T11:22:02.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:01 smithi189 ceph-mon[27694]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:02.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:01 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:01 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:01 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:01 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:02.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:01 smithi189 ceph-mon[27694]: Deploying daemon node-exporter.a on smithi184 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=infra.usagestats.collector t=2024-03-24T11:22:02.1714241Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=provisioning.datasources t=2024-03-24T11:22:02.172557249Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=provisioning.datasources t=2024-03-24T11:22:02.183338468Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=provisioning.alerting t=2024-03-24T11:22:02.193917572Z level=info msg="starting to provision alerting" 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=provisioning.alerting t=2024-03-24T11:22:02.193937261Z level=info msg="finished to provision alerting" 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=ngalert.state.manager t=2024-03-24T11:22:02.194234321Z level=info msg="Warming state cache for startup" 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=ngalert.state.manager t=2024-03-24T11:22:02.194383568Z level=info msg="State cache has been initialized" states=0 duration=148.337µs 2024-03-24T11:22:02.446 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=ticker t=2024-03-24T11:22:02.194484843Z level=info msg=starting first_tick=2024-03-24T11:22:10Z 2024-03-24T11:22:02.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=ngalert.multiorg.alertmanager t=2024-03-24T11:22:02.194533761Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T11:22:02.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=grafanaStorageLogger t=2024-03-24T11:22:02.194825679Z level=info msg="storage starting" 2024-03-24T11:22:02.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=http.server t=2024-03-24T11:22:02.196532686Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T11:22:02.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=sqlstore.transactions t=2024-03-24T11:22:02.207073952Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:22:02.447 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:22:02 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a[38497]: logger=sqlstore.transactions t=2024-03-24T11:22:02.221863647Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T11:22:03.333 INFO:teuthology.orchestra.run.smithi184.stdout:Backend: cephadm 2024-03-24T11:22:03.333 INFO:teuthology.orchestra.run.smithi184.stdout:Available: Yes 2024-03-24T11:22:03.333 INFO:teuthology.orchestra.run.smithi184.stdout:Paused: No 2024-03-24T11:22:04.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:03 smithi186 ceph-mon[27599]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:04.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:03 smithi189 ceph-mon[27694]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:04.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:03 smithi184 ceph-mon[25816]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:04.329 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T11:22:05.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:04 smithi186 ceph-mon[27599]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:05.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:04 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:05.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:04 smithi189 ceph-mon[27694]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:05.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:04 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:05.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:04 smithi184 ceph-mon[25816]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:05.128 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:04 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:06.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:05 smithi184 ceph-mon[25816]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:06.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:05 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:06.040 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:05 smithi184 podman[50921]: 2024-03-24 11:22:05.821276681 +0000 UTC m=+2.922618167 container create f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:06.040 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 podman[50921]: 2024-03-24 11:22:06.039854235 +0000 UTC m=+3.141195689 container init f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:06.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:05 smithi186 ceph-mon[27599]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:06.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:05 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:06.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:05 smithi189 ceph-mon[27694]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:06.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:05 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:06.422 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 podman[50921]: 2024-03-24 11:22:06.045090113 +0000 UTC m=+3.146431557 container start f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:06.422 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.048Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:22:06.422 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.048Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:22:06.422 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:22:06.422 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:22:06.422 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:22:06.423 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.049Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:22:06.424 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:22:06.425 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a[51124]: ts=2024-03-24T11:22:06.050Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 bash[50921]: f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f 2024-03-24T11:22:06.426 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:22:06 smithi184 systemd[1]: Started Ceph node-exporter.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:07.135 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:07.255 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: Deploying daemon node-exporter.b on smithi186 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.256 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:07 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.372 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:07 smithi186 systemd[1]: Starting Ceph node-exporter.b for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:07.372 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:07.372 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.372 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.372 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: Deploying daemon node-exporter.b on smithi186 2024-03-24T11:22:07.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:07 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: Deploying daemon node-exporter.b on smithi186 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:07 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:07.722 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:07 smithi186 bash[39515]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T11:22:07.722 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:07 smithi186 bash[39515]: Getting image source signatures 2024-03-24T11:22:07.723 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:07 smithi186 bash[39515]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T11:22:07.723 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:07 smithi186 bash[39515]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T11:22:09.291 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-mon[27599]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:09.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:09.292 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:07 smithi186 bash[39515]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T11:22:09.292 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:08 smithi186 bash[39515]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:22:09.292 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 bash[39515]: Writing manifest to image destination 2024-03-24T11:22:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:09 smithi184 ceph-mon[25816]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:09.402 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:09 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:09.615 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:09 smithi189 ceph-mon[27694]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:09.616 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:09 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi184 smithi184 running 15s ago 3m - - 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi186 smithi186 running 2s ago 3m - - 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:agent.smithi189 smithi189 running 17s ago 3m - - 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a smithi184 *:9093,9094 running (24s) 15s ago 23s 16.5M - 0.25.0 c8568f914cd2 aa321a68efde 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a smithi186 *:3000 running (9s) 2s ago 8s 57.3M - 9.4.12 a0ea5e3c46d9 f82c1f3be8a7 2024-03-24T11:22:09.641 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x smithi186 *:8443,8765 running (2m) 2s ago 2m 451M - 19.0.0-1667-gdb0330b1 4f325f8440e9 d279fc1c9bb7 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.y smithi184 *:9283,8765 running (4m) 15s ago 4m 497M - 19.0.0-1667-gdb0330b1 4f325f8440e9 85500b4469ea 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.z smithi189 *:8443,8765 running (2m) 17s ago 2m 448M - 19.0.0-1667-gdb0330b1 4f325f8440e9 8b6df1a4dae4 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a smithi184 running (4m) 15s ago 4m 46.6M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 b5a59501fd40 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b smithi186 running (3m) 2s ago 3m 35.8M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 f2cf77d271fd 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c smithi189 running (3m) 17s ago 3m 39.1M 2048M 19.0.0-1667-gdb0330b1 4f325f8440e9 9c795b036f71 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a smithi184 *:9100 starting - - - - 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0 smithi184 running (2m) 15s ago 2m 60.9M 4096M 19.0.0-1667-gdb0330b1 4f325f8440e9 c98d9797e663 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1 smithi186 running (2m) 2s ago 119s 61.6M 4096M 19.0.0-1667-gdb0330b1 4f325f8440e9 8cff19f57678 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2 smithi189 running (94s) 17s ago 93s 50.9M 4096M 19.0.0-1667-gdb0330b1 4f325f8440e9 2221642642a0 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a smithi186 *:9095 running (54s) 2s ago 53s 36.6M - 2.43.0 a07b618ecd1d a38d171830d7 2024-03-24T11:22:09.642 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.a smithi184 *:80 running (75s) 15s ago 75s 77.1M - 19.0.0-1667-gdb0330b1 4f325f8440e9 e7db05ccfb6d 2024-03-24T11:22:09.674 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 podman[39515]: 2024-03-24 11:22:09.341183234 +0000 UTC m=+2.062619982 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:22:09.674 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 podman[39515]: 2024-03-24 11:22:09.406025283 +0000 UTC m=+2.127462029 container create 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:22:09.674 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 podman[39515]: 2024-03-24 11:22:09.591154007 +0000 UTC m=+2.312590762 container init 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:22:09.674 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 podman[39515]: 2024-03-24 11:22:09.596524226 +0000 UTC m=+2.317960971 container start 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.600Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.600Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.600Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.600Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.600Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.600Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:22:09.675 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:22:09.676 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:22:09.676 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:22:09.676 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:22:09.676 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:22:09.676 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:22:09.677 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:22:09.678 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b[39565]: ts=2024-03-24T11:22:09.601Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:22:09.679 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 bash[39515]: 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 2024-03-24T11:22:10.042 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:22:09 smithi186 systemd[1]: Started Ceph node-exporter.b for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:10.863 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T11:22:10.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:10 smithi189 ceph-mon[27694]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:10.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:10 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:10.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:10 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:10.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:10 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:10.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:10 smithi189 ceph-mon[27694]: Deploying daemon node-exporter.c on smithi189 2024-03-24T11:22:10.966 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:10 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:22:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:10.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:10 smithi184 ceph-mon[25816]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:10.966 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:10 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:10.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:10 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:10.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:10 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:10.967 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:10 smithi184 ceph-mon[25816]: Deploying daemon node-exporter.c on smithi189 2024-03-24T11:22:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:10 smithi186 ceph-mon[27599]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:10 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:10 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:10 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:11.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:10 smithi186 ceph-mon[27599]: Deploying daemon node-exporter.c on smithi189 2024-03-24T11:22:11.344 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:11.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:11 smithi184 ceph-mon[25816]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:12.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:11 smithi186 ceph-mon[27599]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:12.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:11 smithi189 ceph-mon[27694]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:13.116 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:12 smithi189 bash[39798]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:agent 3/3 19s ago 49s * 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager ?:9093,9094 1/1 19s ago 73s smithi184=a;count:1 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:grafana ?:3000 1/1 6s ago 71s smithi186=a;count:1 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:mgr 3/3 19s ago 2m smithi184=y;smithi186=x;smithi189=z;count:3 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:mon 3/3 19s ago 3m smithi184:172.21.15.184=a;smithi186:172.21.15.186=b;smithi189:172.21.15.189=c;count:3 2024-03-24T11:22:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter ?:9100 0/3 - 76s smithi184=a;smithi186=b;smithi189=c;count:3 2024-03-24T11:22:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:osd 3 19s ago - 2024-03-24T11:22:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus ?:9095 1/1 6s ago 78s smithi186=a;count:1 2024-03-24T11:22:13.236 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.foo.a ?:80 1/1 19s ago 78s smithi184=foo.a;count:1 2024-03-24T11:22:13.582 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:13 smithi189 bash[39798]: Writing manifest to image destination 2024-03-24T11:22:13.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:13 smithi184 ceph-mon[25816]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:13.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:13 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:13.779 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:13 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:13.780 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:13 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:13.845 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:13 smithi189 podman[39798]: 2024-03-24 11:22:13.638339041 +0000 UTC m=+2.830745094 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T11:22:13.845 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:13 smithi189 podman[39798]: 2024-03-24 11:22:13.845536651 +0000 UTC m=+3.037942730 container create 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-24T11:22:13.846 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:13 smithi189 ceph-mon[27694]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:13.846 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:13 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:13.846 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:13 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:13.846 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:13 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:13 smithi186 ceph-mon[27599]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:14.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:13 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:13 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:13 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 podman[39798]: 2024-03-24 11:22:14.106223303 +0000 UTC m=+3.298629351 container init 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 podman[39798]: 2024-03-24 11:22:14.111834897 +0000 UTC m=+3.304240936 container start 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.114Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.114Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.115Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.115Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T11:22:14.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.115Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.115Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T11:22:14.367 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T11:22:14.368 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T11:22:14.369 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T11:22:14.370 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T11:22:14.371 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T11:22:14.371 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c[40032]: ts=2024-03-24T11:22:14.116Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T11:22:14.371 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 bash[39798]: 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 2024-03-24T11:22:14.371 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:22:14 smithi189 systemd[1]: Started Ceph node-exporter.c for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:14.559 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T11:22:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:14.805 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:22:14.806 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:14 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:22:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:14 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.220 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:22:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:14 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:15.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:22:15.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:22:15.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: Reconfiguring daemon alertmanager.a on smithi184 2024-03-24T11:22:15.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:15 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: Reconfiguring daemon alertmanager.a on smithi184 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:15 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:15.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-24T11:22:15.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T11:22:15.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: Reconfiguring daemon alertmanager.a on smithi184 2024-03-24T11:22:15.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.950 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.950 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:15.950 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:15 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:17 smithi186 ceph-mon[27599]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:17 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:17 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:17 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:17 smithi189 ceph-mon[27694]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:17.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:17 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:17 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:17 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:17 smithi184 ceph-mon[25816]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:17.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:17 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.921 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:17 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:17.922 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:17 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:18.856 INFO:teuthology.orchestra.run.smithi184.stdout:HOST ADDR LABELS STATUS 2024-03-24T11:22:18.856 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 172.21.15.184 2024-03-24T11:22:18.856 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 172.21.15.186 2024-03-24T11:22:18.856 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 172.21.15.189 2024-03-24T11:22:18.856 INFO:teuthology.orchestra.run.smithi184.stdout:3 hosts in cluster 2024-03-24T11:22:19.847 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:19 smithi184 ceph-mon[25816]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:19.847 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:19 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.847 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:19 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.847 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:19 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.848 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:19 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.848 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:19 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.865 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:19 smithi186 ceph-mon[27599]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:19.865 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:19 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.865 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:19 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.865 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:19 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.865 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:19 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.865 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:19 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:19 smithi189 ceph-mon[27694]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:19 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:19 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:19 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:19 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:19.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:19 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:20.168 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T11:22:20.727 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:20 smithi184 ceph-mon[25816]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:20.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:20 smithi189 ceph-mon[27694]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:21.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:20 smithi186 ceph-mon[27599]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:21.087 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:22:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:21.277 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:21.403 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:21 smithi184 systemd[1]: Stopping Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:21.671 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:21 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[48970]: ts=2024-03-24T11:22:21.644Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:22:21.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:21 smithi184 ceph-mon[25816]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:21.671 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:21 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:21 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:21 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.791 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:21 smithi186 ceph-mon[27599]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:21.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:21 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:21 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:21 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:21 smithi189 ceph-mon[27694]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:21.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:21 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:21 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:21.866 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:21 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:22.720 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:22 smithi184 podman[52838]: 2024-03-24 11:22:22.494219435 +0000 UTC m=+0.982425368 container died aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:23.020 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:22 smithi184 podman[52838]: 2024-03-24 11:22:22.811393945 +0000 UTC m=+1.299599876 container cleanup aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:23.020 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:22 smithi184 bash[52838]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:22:23.323 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 podman[52873]: 2024-03-24 11:22:23.215218927 +0000 UTC m=+1.553396392 container remove aa321a68efde26fefc81450bb8235a9e487eca8072c309bb959010bc27fb5851 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:23.324 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 podman[52873]: 2024-03-24 11:22:23.324201296 +0000 UTC m=+1.662378721 volume remove 29b33e1ca44c70017e8ce183dffb2e5a8429f353c5748fc4a045c48da5dbe411 2024-03-24T11:22:23.821 INFO:teuthology.orchestra.run.smithi184.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T11:22:23.821 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT626500CE400BGN 372G No 10s ago Has a FileSystem, LVM detected 2024-03-24T11:22:23.821 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 /dev/nvme1n1 ssd Linux_6f0cdbb475a75095928a 89.4G Yes 10s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 /dev/nvme2n1 ssd Linux_34685be692930456d029 89.4G Yes 10s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 /dev/nvme3n1 ssd Linux_ac26ed7a7a1c4e45d3ca 89.4G Yes 10s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 /dev/nvme4n1 ssd Linux_648a2b1fd71b8245e591 89.4G No 10s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XLJY 931G No 10s ago Has partitions 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 2s ago Has a FileSystem, LVM detected 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 /dev/nvme1n1 ssd Linux_9e7fc2812de554b6985e 89.4G Yes 2s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 /dev/nvme2n1 ssd Linux_016264dd34a73a59e4b4 89.4G Yes 2s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 /dev/nvme3n1 ssd Linux_221fe5394eeda1ce3731 89.4G Yes 2s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 /dev/nvme4n1 ssd Linux_cc71ecd1867a6b0a3ef5 89.4G No 2s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi186 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XTX2 931G No 2s ago Has partitions 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 4s ago Has a FileSystem, LVM detected 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 /dev/nvme1n1 ssd Linux_bcebc7a9d322a8561536 89.4G Yes 4s ago 2024-03-24T11:22:23.822 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 /dev/nvme2n1 ssd Linux_8b4ea99af3db48bd7ece 89.4G Yes 4s ago 2024-03-24T11:22:23.823 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 /dev/nvme3n1 ssd Linux_610a6d81b2400c77931b 89.4G Yes 4s ago 2024-03-24T11:22:23.823 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 /dev/nvme4n1 ssd Linux_492ff378e68fc379c85f 89.4G No 4s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-24T11:22:23.823 INFO:teuthology.orchestra.run.smithi184.stdout:smithi189 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XKLY 931G No 4s ago Has partitions 2024-03-24T11:22:23.865 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:23 smithi189 ceph-mon[27694]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:23.890 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:23 smithi184 ceph-mon[25816]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:23.891 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:22:23.891 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 systemd[1]: Stopped Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:23.891 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 systemd[1]: Starting Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:24.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:23 smithi186 ceph-mon[27599]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:24.163 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 podman[53066]: 2024-03-24 11:22:23.793302301 +0000 UTC m=+0.017692702 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-24T11:22:24.163 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:23 smithi184 podman[53066]: 2024-03-24 11:22:23.925934559 +0000 UTC m=+0.150324951 volume create f4d031f17b7e5d288dd8af46596f07c4c3f9e05e040509f9b0cd61f7f4ea40c9 2024-03-24T11:22:24.164 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 podman[53066]: 2024-03-24 11:22:24.06837606 +0000 UTC m=+0.292766453 container create addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:24.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-mon[25816]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:24.826 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 podman[53066]: 2024-03-24 11:22:24.572168111 +0000 UTC m=+0.796558514 container init addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 podman[53066]: 2024-03-24 11:22:24.578126042 +0000 UTC m=+0.802516434 container start addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.591Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.591Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.592Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.184 port=9094 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.593Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.617Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.617Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.620Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:24.620Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T11:22:24.827 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 bash[53066]: addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 2024-03-24T11:22:24.828 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:24 smithi184 systemd[1]: Started Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:25.041 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:24 smithi186 ceph-mon[27599]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:25.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:24 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:25.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:24 smithi189 ceph-mon[27694]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:22:25.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:24 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:22:25.154 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-24T11:22:25.647 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:25.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:25 smithi186 ceph-mon[27599]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:25.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:25.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:25 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:25.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:25 smithi186 ceph-mon[27599]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:22:26.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:25 smithi184 ceph-mon[25816]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:26.041 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:26.042 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:25 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:26.042 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:25 smithi184 ceph-mon[25816]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:22:26.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:25 smithi189 ceph-mon[27694]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:26.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:26.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:25 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:26.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:25 smithi189 ceph-mon[27694]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T11:22:26.857 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:26 smithi184 ceph-mon[25816]: Reconfiguring daemon prometheus.a on smithi186 2024-03-24T11:22:26.857 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:26 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:26.594Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000620826s 2024-03-24T11:22:27.037 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:26 smithi186 ceph-mon[27599]: Reconfiguring daemon prometheus.a on smithi186 2024-03-24T11:22:27.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:26 smithi189 ceph-mon[27694]: Reconfiguring daemon prometheus.a on smithi186 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: agent 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: agent 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout: host_pattern: '*' 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:21:24.048327Z' 2024-03-24T11:22:27.585 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:16.587933Z' 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout: running: 3 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout: size: 3 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:events: 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:- 2024-03-24T11:21:25.480569Z service:agent [INFO] "service was created" 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: alertmanager 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: alertmanager 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout: count: 1 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi184=a 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.586 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:20:59.601886Z' 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: ports: 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: - 9093 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: - 9094 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: running: 0 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: size: 1 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:events: 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:- 2024-03-24T11:21:46.305845Z service:alertmanager [INFO] "service was created" 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: grafana 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: grafana 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: count: 1 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi186=a 2024-03-24T11:22:27.587 INFO:teuthology.orchestra.run.smithi184.stdout:spec: 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: anonymous_access: true 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: protocol: https 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:21:02.202207Z' 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:21.084180Z' 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: ports: 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: - 3000 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: running: 1 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout: size: 1 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout:events: 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout:- 2024-03-24T11:22:00.744444Z service:grafana [INFO] "service was created" 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: mgr 2024-03-24T11:22:27.588 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: mgr 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: count: 3 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi184=y 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi186=x 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi189=z 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:19:18.713311Z' 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:16.587878Z' 2024-03-24T11:22:27.589 INFO:teuthology.orchestra.run.smithi184.stdout: running: 3 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: size: 3 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: mon 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: mon 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: count: 3 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi184:172.21.15.184=a 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi186:172.21.15.186=b 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi189:172.21.15.189=c 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:18:50.904519Z' 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:16.587758Z' 2024-03-24T11:22:27.590 INFO:teuthology.orchestra.run.smithi184.stdout: running: 3 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: size: 3 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: node-exporter 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: node-exporter 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: count: 3 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi184=a 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi186=b 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi189=c 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:20:56.951616Z' 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:16.588224Z' 2024-03-24T11:22:27.591 INFO:teuthology.orchestra.run.smithi184.stdout: ports: 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout: - 9100 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout: running: 3 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout: size: 3 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:events: 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:- 2024-03-24T11:22:14.280658Z service:node-exporter [INFO] "service was created" 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: osd 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: osd 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:unmanaged: true 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:spec: 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout: filter_logic: AND 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout: objectstore: bluestore 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.592 INFO:teuthology.orchestra.run.smithi184.stdout: container_image_id: 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:16.588007Z' 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: running: 3 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: size: 3 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: prometheus 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: prometheus 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: count: 1 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi186=a 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:20:54.284359Z' 2024-03-24T11:22:27.593 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:21.084106Z' 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout: ports: 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout: - 9095 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout: running: 1 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout: size: 1 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout:events: 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout:- 2024-03-24T11:22:14.291021Z service:prometheus [INFO] "service was created" 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout:--- 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout:service_type: rgw 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout:service_id: foo.a 2024-03-24T11:22:27.594 INFO:teuthology.orchestra.run.smithi184.stdout:service_name: rgw.foo.a 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout:placement: 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: count: 1 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: hosts: 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: - smithi184=foo.a 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout:status: 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: created: '2024-03-24T11:20:54.341275Z' 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: last_refresh: '2024-03-24T11:22:16.588081Z' 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: ports: 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: - 80 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: running: 1 2024-03-24T11:22:27.599 INFO:teuthology.orchestra.run.smithi184.stdout: size: 1 2024-03-24T11:22:28.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:27 smithi186 ceph-mon[27599]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:28.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:27 smithi189 ceph-mon[27694]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:28.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:27 smithi184 ceph-mon[25816]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:28.208 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-24T11:22:28.448 INFO:teuthology.orchestra.run.smithi184.stderr:Inferring config /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T11:22:28.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:28 smithi184 ceph-mon[25816]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T11:22:28.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:28.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:28.850 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:28 smithi184 ceph-mon[25816]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:28 smithi186 ceph-mon[27599]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T11:22:29.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.042 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:28 smithi186 ceph-mon[27599]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:28 smithi189 ceph-mon[27694]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-24T11:22:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.116 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:28 smithi189 ceph-mon[27694]: from='mgr.14415 172.21.15.184:0/2032657070' entity='mgr.y' 2024-03-24T11:22:29.792 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:29 smithi186 ceph-mon[27599]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:30.039 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:29 smithi184 ceph-mon[25816]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:30.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:29 smithi189 ceph-mon[27694]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-24T11:22:30.266 INFO:teuthology.orchestra.run.smithi184.stdout:osd 3 11s ago - 2024-03-24T11:22:30.894 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:30 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: ::ffff:172.21.15.186 - - [24/Mar/2024:11:22:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T11:22:30.922 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T11:22:30.932 INFO:tasks.cephadm:Teardown begin 2024-03-24T11:22:30.932 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:22:30.962 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:22:30.995 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:22:31.033 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T11:22:31.034 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:22:31.048 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:22:31.063 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:22:31.089 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T11:22:31.090 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T11:22:31.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a 2024-03-24T11:22:31.171 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:31 smithi184 systemd[1]: Stopping Ceph mon.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:31.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:31] ENGINE Bus STOPPING 2024-03-24T11:22:31.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:22:31.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:31] ENGINE Bus STOPPED 2024-03-24T11:22:31.671 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:31] ENGINE Bus STARTING 2024-03-24T11:22:31.672 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:31] ENGINE Serving on http://:::9283 2024-03-24T11:22:31.672 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:31] ENGINE Bus STARTED 2024-03-24T11:22:31.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a[25793]: 2024-03-24T11:22:31.400+0000 7f2938b89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:22:31.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:31 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a[25793]: 2024-03-24T11:22:31.400+0000 7f2938b89700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T11:22:31.672 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:31 smithi184 podman[53790]: 2024-03-24 11:22:31.451370668 +0000 UTC m=+0.176232404 container died b5a59501fd4016e04b7d359b2a19f05f0121106fb0b6e53b0fb079e06bd0db07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T11:22:32.038 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:31 smithi186 systemd[1]: Stopping Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:32.073 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:31 smithi184 podman[53790]: 2024-03-24 11:22:31.839168569 +0000 UTC m=+0.564030304 container cleanup b5a59501fd4016e04b7d359b2a19f05f0121106fb0b6e53b0fb079e06bd0db07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:22:32.073 INFO:journalctl@ceph.mon.a.smithi184.stdout:Mar 24 11:22:31 smithi184 bash[53790]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-a 2024-03-24T11:22:32.214 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T11:22:32.259 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:32.259 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T11:22:32.259 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T11:22:32.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.b 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.036Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:22:32.288 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:22:32.289 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:22:32.289 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[35938]: ts=2024-03-24T11:22:32.037Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:22:32.289 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 podman[40659]: 2024-03-24 11:22:32.047218344 +0000 UTC m=+0.154382533 container died a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:32.542 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:32 smithi186 systemd[1]: Stopping Ceph mon.b for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:32.542 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 podman[40659]: 2024-03-24 11:22:32.452943887 +0000 UTC m=+0.560108087 container cleanup a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:32.542 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 bash[40659]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:22:32.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 podman[40678]: 2024-03-24 11:22:32.669330769 +0000 UTC m=+0.621491161 container remove a38d171830d7b714dbe353e3f2d0369f8a680bcffce45b892f2b31b6fbe593be (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:32.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T11:22:32.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 systemd[1]: Stopped Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:32.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.171s CPU time. 2024-03-24T11:22:32.922 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:32 smithi186 systemd[1]: Starting Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:33.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:33 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b[27576]: 2024-03-24T11:22:33.108+0000 7f40e1e63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:22:33.292 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:33 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b[27576]: 2024-03-24T11:22:33.108+0000 7f40e1e63700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:22:33.417 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:33 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:33] ENGINE Bus STOPPING 2024-03-24T11:22:33.418 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:33 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T11:22:33.418 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:33 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:33] ENGINE Bus STOPPED 2024-03-24T11:22:33.418 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:33 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:33] ENGINE Bus STARTING 2024-03-24T11:22:33.418 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:33 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:33] ENGINE Serving on http://:::9283 2024-03-24T11:22:33.418 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:33 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y[26073]: [24/Mar/2024:11:22:33] ENGINE Bus STARTED 2024-03-24T11:22:33.750 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:33 smithi186 podman[40903]: 2024-03-24 11:22:33.456570158 +0000 UTC m=+0.018882552 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T11:22:33.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:33 smithi186 podman[40807]: 2024-03-24 11:22:33.460220454 +0000 UTC m=+0.539439741 container died f2cf77d271fdc846048210e3fc3c477f30e6a65d75a37df00793ce03a9a8531c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:22:33.751 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:33 smithi186 podman[40807]: 2024-03-24 11:22:33.750747814 +0000 UTC m=+0.829967098 container cleanup f2cf77d271fdc846048210e3fc3c477f30e6a65d75a37df00793ce03a9a8531c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:22:34.029 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:33 smithi186 bash[40807]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b 2024-03-24T11:22:34.029 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:33 smithi186 podman[40864]: 2024-03-24 11:22:33.952378665 +0000 UTC m=+0.813451563 container remove f2cf77d271fdc846048210e3fc3c477f30e6a65d75a37df00793ce03a9a8531c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:22:34.610 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 podman[40903]: 2024-03-24 11:22:34.354801504 +0000 UTC m=+0.917113929 container create 9d820d620f84cbbefcccf006be0c1865ef8d4790a34d4c3618e0f1b7cb3d375f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:34.610 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 podman[40903]: 2024-03-24 11:22:34.581386361 +0000 UTC m=+1.143698751 container init 9d820d620f84cbbefcccf006be0c1865ef8d4790a34d4c3618e0f1b7cb3d375f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:34.610 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 podman[40903]: 2024-03-24 11:22:34.586703359 +0000 UTC m=+1.149015746 container start 9d820d620f84cbbefcccf006be0c1865ef8d4790a34d4c3618e0f1b7cb3d375f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:22:34.827 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T11:22:34.874 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:22:34 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:22:34.595Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002455266s 2024-03-24T11:22:34.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:34 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-24T11:22:34.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:34 smithi186 systemd[1]: Stopped Ceph mon.b for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:34.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 24 11:22:34 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.b.service: Consumed 4.361s CPU time. 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.610Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.610Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.610Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.610Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.610Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.612Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.612Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.613Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.613Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.615Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T11:22:34.893 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.615Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.447µs 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.615Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.615Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.616Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.616Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.745µs wal_replay_duration=560.617µs wbl_replay_duration=380ns total_replay_duration=619.89µs 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.616Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.616Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.616Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.634Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.63496ms db_storage=723ns remote_storage=1.003µs web_handler=387ns query_engine=479ns scrape=413.873µs scrape_sd=66.431µs notify=16.566µs notify_sd=14.947µs rules=16.915423ms tracing=6.611µs 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.634Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:22:34.634Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 bash[40903]: 9d820d620f84cbbefcccf006be0c1865ef8d4790a34d4c3618e0f1b7cb3d375f 2024-03-24T11:22:34.894 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:22:34 smithi186 systemd[1]: Started Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:22:34.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:34.951 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T11:22:34.951 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T11:22:34.951 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.c 2024-03-24T11:22:35.245 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 systemd[1]: Stopping Ceph mon.c for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:35.593 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-c[27671]: 2024-03-24T11:22:35.243+0000 7f84bf11a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:22:35.593 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-c[27671]: 2024-03-24T11:22:35.243+0000 7f84bf11a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T11:22:35.593 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 podman[40781]: 2024-03-24 11:22:35.296348114 +0000 UTC m=+0.154183668 container died 9c795b036f71bbb522ae433227ec02e7a7e614cfffaf79b8ad296931caa7e59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:22:35.843 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 podman[40781]: 2024-03-24 11:22:35.675574319 +0000 UTC m=+0.533409888 container cleanup 9c795b036f71bbb522ae433227ec02e7a7e614cfffaf79b8ad296931caa7e59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:22:35.843 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 bash[40781]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-c 2024-03-24T11:22:36.115 INFO:journalctl@ceph.mon.c.smithi189.stdout:Mar 24 11:22:35 smithi189 podman[40800]: 2024-03-24 11:22:35.843371064 +0000 UTC m=+0.544108072 container remove 9c795b036f71bbb522ae433227ec02e7a7e614cfffaf79b8ad296931caa7e59a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T11:22:37.114 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T11:22:37.171 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:37.171 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T11:22:37.171 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T11:22:37.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y 2024-03-24T11:22:37.489 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:37 smithi184 systemd[1]: Stopping Ceph mgr.y for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:37.836 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:37 smithi184 podman[54399]: 2024-03-24 11:22:37.496089718 +0000 UTC m=+0.128030188 container died 85500b4469ea705d3f90bf9af058cf813cd4eebd2d3bdd2a17eae28f4c55abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-24T11:22:38.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:37 smithi184 podman[54399]: 2024-03-24 11:22:37.836607984 +0000 UTC m=+0.468548494 container cleanup 85500b4469ea705d3f90bf9af058cf813cd4eebd2d3bdd2a17eae28f4c55abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b) 2024-03-24T11:22:38.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:37 smithi184 bash[54399]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y 2024-03-24T11:22:38.171 INFO:journalctl@ceph.mgr.y.smithi184.stdout:Mar 24 11:22:38 smithi184 podman[54418]: 2024-03-24 11:22:38.013107109 +0000 UTC m=+0.517233667 container remove 85500b4469ea705d3f90bf9af058cf813cd4eebd2d3bdd2a17eae28f4c55abef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b) 2024-03-24T11:22:38.238 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T11:22:38.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:38.276 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T11:22:38.277 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T11:22:38.277 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.x 2024-03-24T11:22:38.602 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:22:38 smithi186 systemd[1]: Stopping Ceph mgr.x for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:38.860 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:22:38 smithi186 podman[41313]: 2024-03-24 11:22:38.602237842 +0000 UTC m=+0.127518247 container died d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T11:22:39.118 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:22:38 smithi186 podman[41313]: 2024-03-24 11:22:38.926117091 +0000 UTC m=+0.451397506 container cleanup d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T11:22:39.118 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:22:38 smithi186 bash[41313]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x 2024-03-24T11:22:39.368 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:22:39 smithi186 podman[41347]: 2024-03-24 11:22:39.118960738 +0000 UTC m=+0.521136097 container remove d279fc1c9bb76bb1de7f8d83c3c984b71c5fb1c81e83024a9fae79e177e1ef17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T11:22:39.369 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 24 11:22:39 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:22:39.373 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T11:22:39.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:39.414 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T11:22:39.415 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T11:22:39.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.z 2024-03-24T11:22:39.732 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:22:39 smithi189 systemd[1]: Stopping Ceph mgr.z for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:40.043 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:22:39 smithi189 podman[41114]: 2024-03-24 11:22:39.732402758 +0000 UTC m=+0.123610132 container died 8b6df1a4dae4149025f2d6a0bc3e9742ef9c2b801af4ae17632ce11b9fe29170 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T11:22:40.359 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:22:40 smithi189 podman[41114]: 2024-03-24 11:22:40.116720928 +0000 UTC m=+0.507928304 container cleanup 8b6df1a4dae4149025f2d6a0bc3e9742ef9c2b801af4ae17632ce11b9fe29170 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T11:22:40.359 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Mar 24 11:22:40 smithi189 bash[41114]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-mgr-z 2024-03-24T11:22:40.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T11:22:40.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:40.516 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T11:22:40.516 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T11:22:40.517 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.0 2024-03-24T11:22:40.848 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:40 smithi184 systemd[1]: Stopping Ceph osd.0 for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:41.947 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0[38383]: 2024-03-24T11:22:41.682+0000 7fab58798700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:22:41.947 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0[38383]: 2024-03-24T11:22:41.682+0000 7fab58798700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-24T11:22:41.947 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:41 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0[38383]: 2024-03-24T11:22:41.682+0000 7fab58798700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:22:47.421 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:47 smithi184 podman[54590]: 2024-03-24 11:22:47.170050988 +0000 UTC m=+5.941212208 container died c98d9797e6639b3049fab98e18864c5d381eab741c58fb59ac5ad479643c8022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T11:22:47.921 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:47 smithi184 podman[54590]: 2024-03-24 11:22:47.618749458 +0000 UTC m=+6.389910715 container cleanup c98d9797e6639b3049fab98e18864c5d381eab741c58fb59ac5ad479643c8022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T11:22:47.921 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:47 smithi184 bash[54590]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:22:48.290 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:47 smithi184 podman[54730]: 2024-03-24 11:22:47.96159403 +0000 UTC m=+0.792241301 container remove c98d9797e6639b3049fab98e18864c5d381eab741c58fb59ac5ad479643c8022 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:22:48.590 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:48 smithi184 podman[54833]: 2024-03-24 11:22:48.193173103 +0000 UTC m=+0.018686263 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:22:48.590 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:48 smithi184 podman[54833]: 2024-03-24 11:22:48.303930803 +0000 UTC m=+0.129443945 container create 98d77ee6fb36e352f38686343d45b5bc83057cf0057e98a1b2c8a51b2e886b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:22:48.590 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:48 smithi184 podman[54833]: 2024-03-24 11:22:48.506698763 +0000 UTC m=+0.332211917 container init 98d77ee6fb36e352f38686343d45b5bc83057cf0057e98a1b2c8a51b2e886b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T11:22:48.590 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:48 smithi184 podman[54833]: 2024-03-24 11:22:48.512585889 +0000 UTC m=+0.338099033 container start 98d77ee6fb36e352f38686343d45b5bc83057cf0057e98a1b2c8a51b2e886b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T11:22:48.590 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:48 smithi184 podman[54833]: 2024-03-24 11:22:48.589945121 +0000 UTC m=+0.415458266 container attach 98d77ee6fb36e352f38686343d45b5bc83057cf0057e98a1b2c8a51b2e886b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T11:22:48.921 INFO:journalctl@ceph.osd.0.smithi184.stdout:Mar 24 11:22:48 smithi184 podman[54833]: 2024-03-24 11:22:48.798530349 +0000 UTC m=+0.624043492 container died 98d77ee6fb36e352f38686343d45b5bc83057cf0057e98a1b2c8a51b2e886b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T11:22:49.368 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T11:22:49.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:49.410 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T11:22:49.410 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T11:22:49.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.1 2024-03-24T11:22:49.703 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:49 smithi186 systemd[1]: Stopping Ceph osd.1 for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:50.042 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:49 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1[33197]: 2024-03-24T11:22:49.702+0000 7f7ef8faf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:22:50.042 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:49 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1[33197]: 2024-03-24T11:22:49.702+0000 7f7ef8faf700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-24T11:22:50.042 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:49 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1[33197]: 2024-03-24T11:22:49.702+0000 7f7ef8faf700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:22:55.425 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:55 smithi186 podman[41876]: 2024-03-24 11:22:55.151801988 +0000 UTC m=+5.547747588 container died 8cff19f57678fea361ede3b8b0c286edb72331585b3231d64e965a4be947b6ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:22:55.771 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:55 smithi186 podman[41876]: 2024-03-24 11:22:55.533161272 +0000 UTC m=+5.929106879 container cleanup 8cff19f57678fea361ede3b8b0c286edb72331585b3231d64e965a4be947b6ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:22:55.772 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:55 smithi186 bash[41876]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:22:56.542 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:56 smithi186 podman[42017]: 2024-03-24 11:22:56.262263038 +0000 UTC m=+1.109007821 container remove 8cff19f57678fea361ede3b8b0c286edb72331585b3231d64e965a4be947b6ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:22:56.866 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:56 smithi186 podman[42182]: 2024-03-24 11:22:56.514373404 +0000 UTC m=+0.018055673 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:22:56.866 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:56 smithi186 podman[42182]: 2024-03-24 11:22:56.646593176 +0000 UTC m=+0.150275473 container create f8cb8d84b0defc2601b56bbc68edec88fb5563890a696f7327ead8a6b8d93f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T11:22:57.175 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:56 smithi186 podman[42182]: 2024-03-24 11:22:56.865920206 +0000 UTC m=+0.369602469 container init f8cb8d84b0defc2601b56bbc68edec88fb5563890a696f7327ead8a6b8d93f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T11:22:57.175 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:56 smithi186 podman[42182]: 2024-03-24 11:22:56.872062642 +0000 UTC m=+0.375744906 container start f8cb8d84b0defc2601b56bbc68edec88fb5563890a696f7327ead8a6b8d93f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T11:22:57.175 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:56 smithi186 podman[42182]: 2024-03-24 11:22:56.94038704 +0000 UTC m=+0.444069295 container attach f8cb8d84b0defc2601b56bbc68edec88fb5563890a696f7327ead8a6b8d93f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b) 2024-03-24T11:22:57.493 INFO:journalctl@ceph.osd.1.smithi186.stdout:Mar 24 11:22:57 smithi186 podman[42182]: 2024-03-24 11:22:57.175097219 +0000 UTC m=+0.678779473 container died f8cb8d84b0defc2601b56bbc68edec88fb5563890a696f7327ead8a6b8d93f9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-24T11:22:57.871 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T11:22:57.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:22:57.915 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T11:22:57.915 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T11:22:57.915 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.2 2024-03-24T11:22:58.292 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:22:57 smithi189 systemd[1]: Stopping Ceph osd.2 for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:22:58.866 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:22:58 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2[34587]: 2024-03-24T11:22:58.773+0000 7fc55779b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:22:58.866 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:22:58 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2[34587]: 2024-03-24T11:22:58.773+0000 7fc55779b700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-24T11:22:58.866 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:22:58 smithi189 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2[34587]: 2024-03-24T11:22:58.773+0000 7fc55779b700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:23:04.521 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:04 smithi189 podman[41791]: 2024-03-24 11:23:04.271706163 +0000 UTC m=+5.597911514 container died 2221642642a07a36539dff31b090f3f8b31c7044fd9001d6267dce84e881bc06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0) 2024-03-24T11:23:04.779 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:04 smithi189 podman[41791]: 2024-03-24 11:23:04.620411634 +0000 UTC m=+5.946616976 container cleanup 2221642642a07a36539dff31b090f3f8b31c7044fd9001d6267dce84e881bc06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-24T11:23:04.779 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:04 smithi189 bash[41791]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:23:05.116 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:04 smithi189 podman[41813]: 2024-03-24 11:23:04.779758456 +0000 UTC m=+0.503427894 container remove 2221642642a07a36539dff31b090f3f8b31c7044fd9001d6267dce84e881bc06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T11:23:05.491 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:05 smithi189 podman[41911]: 2024-03-24 11:23:05.089453355 +0000 UTC m=+0.017284146 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T11:23:05.491 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:05 smithi189 podman[41911]: 2024-03-24 11:23:05.197052976 +0000 UTC m=+0.124883768 container create 124d60a6c42921d76b83311d8a73decaa724ab349585d7872654847a831490c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:23:05.491 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:05 smithi189 podman[41911]: 2024-03-24 11:23:05.40789403 +0000 UTC m=+0.335724834 container init 124d60a6c42921d76b83311d8a73decaa724ab349585d7872654847a831490c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:23:05.491 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:05 smithi189 podman[41911]: 2024-03-24 11:23:05.413494287 +0000 UTC m=+0.341325079 container start 124d60a6c42921d76b83311d8a73decaa724ab349585d7872654847a831490c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:23:05.492 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:05 smithi189 podman[41911]: 2024-03-24 11:23:05.491103977 +0000 UTC m=+0.418934776 container attach 124d60a6c42921d76b83311d8a73decaa724ab349585d7872654847a831490c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:23:05.866 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:05 smithi189 podman[41911]: 2024-03-24 11:23:05.699737565 +0000 UTC m=+0.627568350 container died 124d60a6c42921d76b83311d8a73decaa724ab349585d7872654847a831490c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:23:06.244 INFO:journalctl@ceph.osd.2.smithi189.stdout:Mar 24 11:23:06 smithi189 podman[41957]: 2024-03-24 11:23:06.235709983 +0000 UTC m=+0.531479744 container remove 124d60a6c42921d76b83311d8a73decaa724ab349585d7872654847a831490c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b) 2024-03-24T11:23:06.250 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T11:23:06.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:23:06.291 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T11:23:06.291 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T11:23:06.292 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T11:23:06.603 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:06 smithi184 systemd[1]: Stopping Ceph rgw.foo.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:06.921 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a[41586]: 2024-03-24T11:23:06.600+0000 7efbf3e32700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:23:06.921 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:06 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a[41586]: 2024-03-24T11:23:06.601+0000 7efc026a2ac0 -1 shutting down 2024-03-24T11:23:16.733 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:16 smithi184 bash[55637]: time="2024-03-24T11:23:16Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T11:23:17.053 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:16 smithi184 podman[55637]: 2024-03-24 11:23:16.733067798 +0000 UTC m=+10.234678296 container died e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T11:23:17.372 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:17 smithi184 podman[55637]: 2024-03-24 11:23:17.052628664 +0000 UTC m=+10.554239164 container cleanup e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T11:23:17.372 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:17 smithi184 bash[55637]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T11:23:17.372 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:17 smithi184 podman[56190]: 2024-03-24 11:23:17.228830685 +0000 UTC m=+0.597611854 container remove e7db05ccfb6d778c457ce63fe2269b7c222c6a133e7313a3f9e1013fe56c5cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T11:23:17.372 INFO:journalctl@ceph.rgw.foo.a.smithi184.stdout:Mar 24 11:23:17 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T11:23:18.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T11:23:18.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:23:18.183 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T11:23:18.184 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T11:23:18.184 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T11:23:18.468 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 systemd[1]: Stopping Ceph prometheus.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:18.469 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:23:18.469 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T11:23:18.469 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T11:23:18.469 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T11:23:18.469 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T11:23:18.470 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T11:23:18.470 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T11:23:18.470 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.467Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T11:23:18.470 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.468Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T11:23:18.470 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.468Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T11:23:18.470 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.468Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T11:23:18.471 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a[40959]: ts=2024-03-24T11:23:18.468Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T11:23:18.731 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 podman[43070]: 2024-03-24 11:23:18.478124872 +0000 UTC m=+0.099109110 container died 9d820d620f84cbbefcccf006be0c1865ef8d4790a34d4c3618e0f1b7cb3d375f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:19.042 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 podman[43070]: 2024-03-24 11:23:18.822413327 +0000 UTC m=+0.443397571 container cleanup 9d820d620f84cbbefcccf006be0c1865ef8d4790a34d4c3618e0f1b7cb3d375f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:19.042 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 24 11:23:18 smithi186 bash[43070]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T11:23:19.237 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T11:23:19.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:23:19.275 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T11:23:19.275 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:23:19.457 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:23:20.754 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:20 smithi184 systemd[1]: Stopping Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:20.754 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:20 smithi184 ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a[53094]: ts=2024-03-24T11:23:20.733Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T11:23:21.066 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:20 smithi184 podman[56674]: 2024-03-24 11:23:20.754057708 +0000 UTC m=+0.130600569 container died addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:21.421 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:21 smithi184 podman[56674]: 2024-03-24 11:23:21.214050341 +0000 UTC m=+0.590593341 container cleanup addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:21.421 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:21 smithi184 bash[56674]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T11:23:21.762 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:21 smithi184 podman[56692]: 2024-03-24 11:23:21.464973258 +0000 UTC m=+0.710339351 container remove addee6d74834bb60f11d11a6a4b2682f2a7694e22c88ef7c4b6d3529784908d4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:21.763 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:21 smithi184 podman[56692]: 2024-03-24 11:23:21.573513846 +0000 UTC m=+0.818879941 volume remove f4d031f17b7e5d288dd8af46596f07c4c3f9e05e040509f9b0cd61f7f4ea40c9 2024-03-24T11:23:22.171 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:21 smithi184 systemd[1]: Stopping Ceph node-exporter.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:22.171 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:21 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T11:23:22.171 INFO:journalctl@ceph.alertmanager.a.smithi184.stdout:Mar 24 11:23:21 smithi184 systemd[1]: Stopped Ceph alertmanager.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:23:22.584 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 podman[56810]: 2024-03-24 11:23:22.214840318 +0000 UTC m=+0.098360792 container died f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:22.585 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 podman[56810]: 2024-03-24 11:23:22.585042437 +0000 UTC m=+0.468562910 container cleanup f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:22.912 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 bash[56810]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T11:23:22.913 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 podman[56830]: 2024-03-24 11:23:22.802706172 +0000 UTC m=+0.587992912 container remove f6b5d7cecec022a6deb4aa064fc0d52eea74a31cf958ba013b5f8ca7ea679f1f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T11:23:22.913 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:23:23.335 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T11:23:23.336 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 systemd[1]: Stopped Ceph node-exporter.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:23:23.336 INFO:journalctl@ceph.node-exporter.a.smithi184.stdout:Mar 24 11:23:22 smithi184 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 1.918s CPU time. 2024-03-24T11:23:23.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:23:23.768 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:23:25.042 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:24 smithi186 systemd[1]: Stopping Ceph grafana.a for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:25.436 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 podman[43458]: 2024-03-24 11:23:25.156452154 +0000 UTC m=+0.114797371 container died f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, description=Ceph Grafana Container, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.buildah.version=1.32.0) 2024-03-24T11:23:25.768 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 podman[43458]: 2024-03-24 11:23:25.517697542 +0000 UTC m=+0.476042758 container cleanup f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=) 2024-03-24T11:23:25.768 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 bash[43458]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a 2024-03-24T11:23:26.042 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 podman[43477]: 2024-03-24 11:23:25.768633759 +0000 UTC m=+0.615133812 container remove f82c1f3be8a7244857225fbd341b9d66e9bed0a72a26c2e7f75c2c38e1f7fdf7 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, io.openshift.tags=base rhel8, architecture=x86_64, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, distribution-scope=public, maintainer=Paul Cuzner , name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897) 2024-03-24T11:23:26.042 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:23:26.042 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T11:23:26.042 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 systemd[1]: Stopped Ceph grafana.a for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:23:26.042 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 24 11:23:25 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@grafana.a.service: Consumed 3.929s CPU time. 2024-03-24T11:23:26.393 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:26 smithi186 systemd[1]: Stopping Ceph node-exporter.b for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:26.663 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:26 smithi186 podman[43595]: 2024-03-24 11:23:26.393384602 +0000 UTC m=+0.123123649 container died 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:23:26.988 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:26 smithi186 podman[43595]: 2024-03-24 11:23:26.75444017 +0000 UTC m=+0.484179209 container cleanup 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:23:26.988 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:26 smithi186 bash[43595]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T11:23:27.274 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:26 smithi186 podman[43613]: 2024-03-24 11:23:26.988689639 +0000 UTC m=+0.596822446 container remove 152de5133051009835f12c17c5f08b8c5f8b846b83d905a93de733aef3c9f6b5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T11:23:27.274 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:26 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:23:27.274 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:27 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T11:23:27.274 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:27 smithi186 systemd[1]: Stopped Ceph node-exporter.b for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:23:27.274 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 24 11:23:27 smithi186 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 1.823s CPU time. 2024-03-24T11:23:27.843 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T11:23:28.038 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:23:29.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 systemd[1]: Stopping Ceph node-exporter.c for f43a2692-e9cf-11ee-95cd-87774f69a715... 2024-03-24T11:23:29.366 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 podman[42777]: 2024-03-24 11:23:29.240373956 +0000 UTC m=+0.086741172 container died 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-24T11:23:29.852 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 podman[42777]: 2024-03-24 11:23:29.643296075 +0000 UTC m=+0.489663291 container cleanup 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-24T11:23:29.852 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 bash[42777]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c 2024-03-24T11:23:30.116 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 podman[42796]: 2024-03-24 11:23:29.852410213 +0000 UTC m=+0.616150242 container remove 384029817cc0f27cebd85f691ce9cf71d78636a7cf503224cd6b4ee813d2e2c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f43a2692-e9cf-11ee-95cd-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-24T11:23:30.116 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-24T11:23:30.116 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-24T11:23:30.116 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 systemd[1]: Stopped Ceph node-exporter.c for f43a2692-e9cf-11ee-95cd-87774f69a715. 2024-03-24T11:23:30.116 INFO:journalctl@ceph.node-exporter.c.smithi189.stdout:Mar 24 11:23:29 smithi189 systemd[1]: ceph-f43a2692-e9cf-11ee-95cd-87774f69a715@node-exporter.c.service: Consumed 1.853s CPU time. 2024-03-24T11:23:30.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:23:30.654 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:23:30.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:23:30.719 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T11:23:30.720 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi184/crash 2024-03-24T11:23:30.721 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:23:30.760 INFO:teuthology.orchestra.run.smithi184.stderr:tar: /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:23:30.760 INFO:teuthology.orchestra.run.smithi184.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:23:30.762 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi186/crash 2024-03-24T11:23:30.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:23:30.813 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:23:30.813 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:23:30.815 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi189/crash 2024-03-24T11:23:30.816 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:23:30.856 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T11:23:30.856 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-03-24T11:23:30.858 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T11:23:30.858 DEBUG:teuthology.orchestra.run.smithi184:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:23:30.902 INFO:teuthology.orchestra.run.smithi184.stdout:2024-03-24T11:19:10.634390+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:23:30.902 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T11:23:30.903 DEBUG:teuthology.orchestra.run.smithi184:> sudo egrep '\[SEC\]' /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:23:30.972 DEBUG:teuthology.orchestra.run.smithi184:> sudo egrep '\[ERR\]' /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:23:31.041 DEBUG:teuthology.orchestra.run.smithi184:> sudo egrep '\[WRN\]' /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-24T11:23:31.110 INFO:teuthology.orchestra.run.smithi184.stdout:2024-03-24T11:19:10.634390+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T11:23:31.110 INFO:tasks.cephadm:Compressing logs... 2024-03-24T11:23:31.110 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:23:31.153 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:23:31.155 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:: No such file or directory 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T11:23:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T11:23:31.181 INFO:teuthology.orchestra.run.smithi184.stderr: 57.3% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T11:23:31.181 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:23:31.181 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.y.log: /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:23:31.181 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:23:31.181 INFO:teuthology.orchestra.run.smithi184.stderr: 83.1% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:23:31.181 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log: 90.1% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:23:31.182 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log: 85.2% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:23:31.182 INFO:teuthology.orchestra.run.smithi184.stderr: 83.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:23:31.183 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:23:31.183 INFO:teuthology.orchestra.run.smithi189.stderr:: No such file or directory 2024-03-24T11:23:31.183 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:23:31.183 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T11:23:31.184 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:23:31.184 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:23:31.187 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T11:23:31.187 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log: /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:23:31.187 INFO:teuthology.orchestra.run.smithi189.stderr: 82.1% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:23:31.187 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.2.log: 90.2% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:23:31.188 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T11:23:31.188 INFO:teuthology.orchestra.run.smithi189.stderr: 85.0% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:23:31.188 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log: 78.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:23:31.188 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:23:31.188 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr: 90.2% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr: 82.1% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T11:23:31.189 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.b.log: 84.8% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:23:31.190 INFO:teuthology.orchestra.run.smithi186.stderr: 77.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:23:31.190 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T11:23:31.191 INFO:teuthology.orchestra.run.smithi186.stderr: 90.3% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T11:23:31.201 INFO:teuthology.orchestra.run.smithi189.stderr: 90.2% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:23:31.203 INFO:teuthology.orchestra.run.smithi186.stderr: 90.2% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:23:31.206 INFO:teuthology.orchestra.run.smithi184.stderr: 90.1% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:23:31.253 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T11:23:31.267 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T11:23:31.305 INFO:teuthology.orchestra.run.smithi184.stderr: 88.1% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T11:23:31.437 INFO:teuthology.orchestra.run.smithi184.stderr: 91.6% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T11:23:31.456 INFO:teuthology.orchestra.run.smithi184.stderr: 94.3% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T11:23:31.458 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-03-24T11:23:31.458 INFO:teuthology.orchestra.run.smithi184.stderr:real 0m0.294s 2024-03-24T11:23:31.458 INFO:teuthology.orchestra.run.smithi184.stderr:user 0m0.694s 2024-03-24T11:23:31.458 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m0.035s 2024-03-24T11:23:31.474 INFO:teuthology.orchestra.run.smithi189.stderr: 94.3% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T11:23:31.476 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-24T11:23:31.476 INFO:teuthology.orchestra.run.smithi189.stderr:real 0m0.306s 2024-03-24T11:23:31.476 INFO:teuthology.orchestra.run.smithi189.stderr:user 0m0.393s 2024-03-24T11:23:31.476 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m0.035s 2024-03-24T11:23:31.502 INFO:teuthology.orchestra.run.smithi186.stderr: 94.3% -- replaced with /var/log/ceph/f43a2692-e9cf-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T11:23:31.504 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-24T11:23:31.504 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m0.334s 2024-03-24T11:23:31.505 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m0.412s 2024-03-24T11:23:31.505 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.036s 2024-03-24T11:23:31.505 INFO:tasks.cephadm:Archiving logs... 2024-03-24T11:23:31.505 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi184/log 2024-03-24T11:23:31.506 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:23:31.641 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi186/log 2024-03-24T11:23:31.642 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:23:31.739 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi189/log 2024-03-24T11:23:31.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:23:31.828 INFO:tasks.cephadm:Removing cluster... 2024-03-24T11:23:31.828 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --force 2024-03-24T11:23:32.011 INFO:teuthology.orchestra.run.smithi184.stdout:Deleting cluster with fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:23:32.127 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --force 2024-03-24T11:23:32.309 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:23:32.424 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f43a2692-e9cf-11ee-95cd-87774f69a715 --force 2024-03-24T11:23:32.607 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: f43a2692-e9cf-11ee-95cd-87774f69a715 2024-03-24T11:23:32.721 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T11:23:32.721 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:23:32.738 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:23:32.755 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T11:23:32.770 INFO:tasks.cephadm:Teardown complete 2024-03-24T11:23:32.772 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-24T11:23:32.783 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi184:/dev/vg_nvme/lv_1... 2024-03-24T11:23:32.784 DEBUG:teuthology.orchestra.run.smithi184:> sudo nvme disconnect -n lv_1 2024-03-24T11:23:32.939 INFO:teuthology.orchestra.run.smithi184.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T11:23:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:32.941 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi184:/dev/vg_nvme/lv_2... 2024-03-24T11:23:32.941 DEBUG:teuthology.orchestra.run.smithi184:> sudo nvme disconnect -n lv_2 2024-03-24T11:23:33.091 INFO:teuthology.orchestra.run.smithi184.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T11:23:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.093 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi184:/dev/vg_nvme/lv_3... 2024-03-24T11:23:33.093 DEBUG:teuthology.orchestra.run.smithi184:> sudo nvme disconnect -n lv_3 2024-03-24T11:23:33.242 INFO:teuthology.orchestra.run.smithi184.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T11:23:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.244 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi184:/dev/vg_nvme/lv_4... 2024-03-24T11:23:33.244 DEBUG:teuthology.orchestra.run.smithi184:> sudo nvme disconnect -n lv_4 2024-03-24T11:23:33.402 INFO:teuthology.orchestra.run.smithi184.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T11:23:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.405 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-03-24T11:23:33.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/scratch_devs 2024-03-24T11:23:33.435 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi186:/dev/vg_nvme/lv_1... 2024-03-24T11:23:33.435 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme disconnect -n lv_1 2024-03-24T11:23:33.581 INFO:teuthology.orchestra.run.smithi186.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T11:23:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.584 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi186:/dev/vg_nvme/lv_2... 2024-03-24T11:23:33.584 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme disconnect -n lv_2 2024-03-24T11:23:33.737 INFO:teuthology.orchestra.run.smithi186.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T11:23:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.738 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi186:/dev/vg_nvme/lv_3... 2024-03-24T11:23:33.738 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme disconnect -n lv_3 2024-03-24T11:23:33.884 INFO:teuthology.orchestra.run.smithi186.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T11:23:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:33.886 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi186:/dev/vg_nvme/lv_4... 2024-03-24T11:23:33.886 DEBUG:teuthology.orchestra.run.smithi186:> sudo nvme disconnect -n lv_4 2024-03-24T11:23:34.030 INFO:teuthology.orchestra.run.smithi186.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T11:23:34.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.032 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-24T11:23:34.032 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/scratch_devs 2024-03-24T11:23:34.061 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_1... 2024-03-24T11:23:34.061 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_1 2024-03-24T11:23:34.204 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-24T11:23:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.206 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_2... 2024-03-24T11:23:34.206 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_2 2024-03-24T11:23:34.336 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-24T11:23:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.338 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_3... 2024-03-24T11:23:34.339 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_3 2024-03-24T11:23:34.495 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-24T11:23:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.497 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi189:/dev/vg_nvme/lv_4... 2024-03-24T11:23:34.497 DEBUG:teuthology.orchestra.run.smithi189:> sudo nvme disconnect -n lv_4 2024-03-24T11:23:34.638 INFO:teuthology.orchestra.run.smithi189.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-24T11:23:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:34.640 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T11:23:34.640 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/scratch_devs 2024-03-24T11:23:34.669 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T11:23:34.679 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T11:23:34.679 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:23:34.681 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:23:34.684 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:23:34.695 INFO:teuthology.orchestra.run.smithi184.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:23:34.698 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:23:34.700 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +669us[ +669us] +/- 84ms 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -119us[ -118us] +/- 78ms 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -551us[ -550us] +/- 85ms 2024-03-24T11:23:34.701 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +686us[ +709us] +/- 84ms 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -122us[ -100us] +/- 78ms 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -545us[ -523us] +/- 85ms 2024-03-24T11:23:34.703 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:23:34.725 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:23:34.729 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:23:34.729 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-24T11:23:34.729 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +682us[ +682us] +/- 84ms 2024-03-24T11:23:34.729 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -120us[ -116us] +/- 78ms 2024-03-24T11:23:34.729 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -554us[ -551us] +/- 85ms 2024-03-24T11:23:34.729 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:23:34.730 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T11:23:34.739 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T11:23:34.740 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T11:23:34.749 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:23:34.828 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:23:34.903 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:23:34.977 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:23:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.009 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:23:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.041 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:23:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:35.073 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T11:23:35.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T11:23:35.092 INFO:teuthology.task.internal:Duration was 752.000719 seconds 2024-03-24T11:23:35.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T11:23:35.103 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T11:23:35.103 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:23:35.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:23:35.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:23:35.149 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:23:35.152 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:23:35.157 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:23:35.582 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T11:23:35.582 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-03-24T11:23:35.583 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:23:35.603 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-03-24T11:23:35.603 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:23:35.627 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T11:23:35.627 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:23:35.653 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T11:23:35.653 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:23:35.656 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:23:35.669 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:23:35.723 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T11:23:35.723 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:23:35.725 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:23:35.739 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:23:35.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T11:23:35.924 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T11:23:35.924 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:23:35.951 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:23:35.978 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:23:36.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T11:23:36.067 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:36.069 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:36.072 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:36.092 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-03-24T11:23:36.095 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-03-24T11:23:36.097 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-03-24T11:23:36.120 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.174 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.190 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:23:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:23:36.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T11:23:36.217 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T11:23:36.218 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi184 2024-03-24T11:23:36.218 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:23:36.261 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi186 2024-03-24T11:23:36.261 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:23:36.299 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895/remote/smithi189 2024-03-24T11:23:36.299 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:23:36.337 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T11:23:36.338 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:23:36.341 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:23:36.343 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:23:36.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T11:23:36.399 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T11:23:36.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T11:23:36.527 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T11:23:36.527 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:23:36.530 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:23:36.532 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:23:36.542 INFO:teuthology.orchestra.run.smithi184.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:23 /home/ubuntu/cephtest 2024-03-24T11:23:36.544 INFO:teuthology.orchestra.run.smithi186.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:23 /home/ubuntu/cephtest 2024-03-24T11:23:36.546 INFO:teuthology.orchestra.run.smithi189.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:23 /home/ubuntu/cephtest 2024-03-24T11:23:36.549 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T11:23:36.559 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T11:23:36.581 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T11:23:36.608 DEBUG:teuthology.nuke:shortname: smithi184 2024-03-24T11:23:36.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:23:36.636 DEBUG:teuthology.nuke:shortname: smithi186 2024-03-24T11:23:36.636 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:23:36.642 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:01:20.713104', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiEBlVUeWGsLpzLiMTCPg7Pe57C5FcD2O5TVEBkD4nHRg1g/VXm5ITHsaJLFdW4Z4Capwzb8tdTOKoKX9zqEGs='} 2024-03-24T11:23:36.662 DEBUG:teuthology.nuke:shortname: smithi189 2024-03-24T11:23:36.662 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:23:36.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:01:20.714262', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7oM0a+vJpqRFpA/47Ap6v19P8bGX4Lz2KLOJwTJ4DXdB5+F7m+AyUyChWOEuv6FBJ4zNdfWD2+g9cu2Sm5kcY='} 2024-03-24T11:23:36.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 11:01:20.711687', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBML+x12VA0DdZ+ylW/KTlOj+3dIz2f/aTzdHblLXpubYJB+PVN+rWsUENRFARcw/JSPirxjvRK07har/RzmlnAE='} 2024-03-24T11:23:36.702 INFO:teuthology.orchestra.console.smithi184:Power off 2024-03-24T11:23:36.702 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:23:36.727 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-03-24T11:23:36.727 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:36.752 INFO:teuthology.orchestra.console.smithi189:Power off 2024-03-24T11:23:36.753 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:23:36.767 INFO:teuthology.orchestra.console.smithi186:Power off 2024-03-24T11:23:36.767 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:23:36.780 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-03-24T11:23:36.780 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:36.789 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-24T11:23:36.798 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-03-24T11:23:36.798 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:36.808 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T11:23:36.821 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-24T11:23:40.790 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:40.809 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:40.822 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:40.900 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-24T11:23:40.919 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T11:23:40.932 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-24T11:23:44.902 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:44.920 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:44.933 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:45.014 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-24T11:23:45.030 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T11:23:45.043 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-24T11:23:49.018 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:49.031 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:49.043 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:23:49.130 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-24T11:23:49.143 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-03-24T11:23:49.154 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-03-24T11:23:49.232 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-03-24T11:23:49.244 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-03-24T11:23:49.255 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-03-24T11:23:49.359 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} duration: 752.0007193088531 failure_reason: '"2024-03-24T11:19:10.634390+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-24T11:23:49.360 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:23:49.451 INFO:teuthology.run:FAIL